Rafał Pydyniak - Liferay Expert Personal Blog

Fixing Liferay GA132 Bug: Client Extensions Being Removed After Modification

Wed, 14 May 2025 00:00:00 GMT

The Problem: Disappearing Client Extensions in Liferay GA132

If you've recently upgraded to Liferay GA132 and noticed that your Client Extensions (CX) keep disappearing from pages after modification, you're not alone. I've encountered this frustrating issue and traced it back to a specific problem in the platform's code.

The issue originates in the CETConfigurationFactory.java, which contains code that removes Client Extension entries from pages after each deactivation and modification.

The problematic fragment:

_clientExtensionEntryRelLocalService.
    deleteClientExtensionEntryRels(
        companyId, _cet.getExternalReferenceCode());

This code is triggered when a Client Extension is modified or deactivated, causing all relations to pages to be removed. While this issue definitely affects Global JS Client Extensions, it likely impacts other types as well, such as Global CSS and potentially others.

When looking at the codebase, it appears this behavior was introduced in GA132, though it's possible it could exist in earlier versions that I haven't tested.

The Impact: Development Frustration and Production Risks

For developers, this bug creates a significant productivity issue. Every time you modify and deploy your Client Extension, you'll need to manually re-add it to all the pages where it was previously used. In a development environment where frequent changes are the norm, this becomes an enormous time sink.

More concerning is the potential impact on production environments. If a developer or administrator modifies a Client Extension in production and forgets to re-add it to all the necessary pages, this could result in broken functionality for end users. Since there's no warning or notification about the removed relations, this type of error could easily go unnoticed until reported by users - creating a poor user experience and potentially affecting business operations.

The most frustrating part is that there's no built-in configuration option to disable this behavior. Looking at the code structure, a custom implementation is necessary to solve the problem.

Official Bug Status

I did some research and I've discovered that this issue has already been reported in the Liferay issue tracking system as LPD-50051, and a fix has been implemented in a pull request.

However, there's an important consideration for Community Edition (CE) users: since CE doesn't receive regular patches, it could be months before this fix makes its way into the next official CE release. If you're using CE and encountering this issue, you'll likely need to implement the workaround described in this article until an updated version becomes available.

Enterprise subscribers may receive the fix sooner through regular patches, but even then, the timeline for patch delivery can vary. The custom implementation described here provides an immediate solution while waiting for the official fix.

Solutions: Creating a Custom Implementation

If you, just like me, cannot wait for next version then you need to figure out a solution.

Option 1: Create a Custom ServiceWrapper

You could create a custom clientExtensionEntryRelLocalServiceWrapper that checks if the delete request is coming from the CETConfigurationFactory class and, if so, prevents the deletion.

While this approach would work, it requires more complex logic to identify the origin of the delete request, making it potentially fragile if internal implementations change.

Option 2: Replace the CETConfigurationFactory (Recommended)

A cleaner approach is to provide a custom implementation of the CETConfigurationFactory itself, where we remove the problematic code, and disable the original factory.

This is the approach I recommend and will detail below.

Implementation Guide: Replacing the CETConfigurationFactory

Here's how to implement the solution:

Step 1: Create a Custom CETConfigurationFactory

Create a custom implementation of CETConfigurationFactory where you remove the problematic code. Essentially, this will be a copy of the original class with the offending lines removed or commented out.

The modified version of the modified method would look like this:

@Modified
protected void modified(Map<String, Object> properties) throws Exception {
   _properties = properties;

   String externalReferenceCode = _getExternalReferenceCode(properties);

   if (_log.isDebugEnabled()) {
      _log.debug(
         StringBundler.concat(
            "Modifying client extension ", externalReferenceCode,
            "with properties:\n", MapUtil.toString(properties)));
   }
   else if (_log.isInfoEnabled()) {
      _log.info("Modifying client extension " + externalReferenceCode);
   }

   ConfigurationFactoryUtil.executeAsCompany(
      _companyLocalService, properties,
      companyId -> {
         try {
            if (_log.isInfoEnabled()) {
               _log.info(
                  StringBundler.concat(
                     "Deleting CET for client extension ",
                     externalReferenceCode, " and company ",
                     companyId));
            }

            _cetManager.deleteCET(_cet);

            if (_log.isInfoEnabled()) {
               _log.info(
                  StringBundler.concat(
                     "Adding CET for client extension ",
                     externalReferenceCode, " and company ",
                     companyId));
            }

            _cet = _cetManager.addCET(
               ConfigurableUtil.createConfigurable(
                  CETConfiguration.class, properties),
               companyId, externalReferenceCode);

            if (_log.isInfoEnabled()) {
               _log.info(
                  StringBundler.concat(
                     "Deleting client extension entry relations ",
                     "for client extension ", externalReferenceCode,
                     " and company ", companyId));
            }
//          commented out code
//          _clientExtensionEntryRelLocalService.
//             deleteClientExtensionEntryRels(
//                companyId, _cet.getExternalReferenceCode());

            if (_log.isInfoEnabled()) {
               _log.info(
                  StringBundler.concat(
                     "Adding client extension entry relations for ",
                     "client extension ", externalReferenceCode,
                     " and company ", companyId));
            }

            if (Objects.equals(
                  _cet.getType(),
                  ClientExtensionEntryConstants.TYPE_THEME_CSS)) {

               _addControlPanelThemeCSSClientExtensionEntryRel(
                  companyId);
            }
         }
         catch (Exception exception) {
            _log.error(
               StringBundler.concat(
                  "Unable to modify client extension ",
                  externalReferenceCode, " for company ", companyId),
               exception);

            throw exception;
         }
      });
}

Important: You need to make the same change in the @Deactivate method as well, since both methods contain the problematic code that removes relations.

Step 2: Disable the Original Factory

To disable the original implementation, create a configuration file at:

osgi/configs/com.liferay.portal.component.blacklist.internal.configuration.ComponentBlacklistConfiguration.config

With the following content:

blacklistComponentNames=[ \
  "com.liferay.client.extension.type.internal.configuration.CETConfigurationFactory" ,\
  ]

This will prevent the original component from being activated, allowing your custom implementation to take over.

Please note if you have such file already: simply add a new line there, no need to create anything new.

But What About Cleanup?

You might be wondering: "If we're preventing the deletion of these relations, what happens when we actually want to remove a Client Extension entirely?"

The good news is that this shouldn't cause any issues. When a CET (Client Extension Type) is removed, references to it become orphaned. Even if the ClientExtensionEntryRel entries remain in the database, they won't affect anything since the system checks for a valid CET before trying to load or use a Client Extension.

In other words, with the CET removed, the page won't try to load the Client Extension anyway, making the cleanup of relations unnecessary for normal operation.

Conclusion

This bug in Liferay GA132 can be quite frustrating for developers working with Client Extensions, but with the custom implementation described above, you can prevent the automatic removal of Client Extensions from pages.

The solution is straightforward to implement and shouldn't cause any issues with the normal operation of your Liferay instance. By replacing the problematic factory class, you ensure that your Client Extensions remain where you put them, even after modifications or redeployments.

Note: This solution has been tested in specific project environment. As always, thoroughly test any modifications in a staging environment before applying them to production.

Resolving Mockito Errors After JDK Upgrades in Liferay Portal

Wed, 30 Apr 2025 00:00:00 GMT

As everyone knows upgrading Liferay Portal or migrating to a newer JDK version (such as JDK 21) can be a pain. Developers might encounter unexpected errors that can slow down development and testing processes. In this article, we'll explore a specific issue related to Mockito and provide a simple solution.

The Problem: Mockito Errors After JDK Upgrade

If you've recently upgraded your Liferay environment to use JDK 21 (or possibly JDK11 or JDK17), you might encounter test failures with error messages similar to:

Caused by: java.lang.reflect.InaccessibleObjectException: Unable to make field static final java.lang.invoke.MethodHandles$Lookup 
java.lang.invoke.MethodHandles$Lookup.IMPL_LOOKUP accessible: module java.base does not "opens java.lang.invoke" to unnamed module

These errors occur because newer JDK versions have implemented a stronger module system (Project Jigsaw) that restricts access to internal JDK packages. Mockito relies on accessing these internal classes for its functionality.

Understanding the Root Cause

The Java Platform Module System (JPMS) introduced in Java 9 enforces stricter encapsulation of internal Java APIs. Libraries like Mockito that use reflection to access internal JDK classes need explicit permission to do so in the form of --add-opens directives.

In our specific case, Mockito needs access to:

java.lang package from the java.base module
java.lang.invoke package from the java.base module

The Solution: Adding JVM Arguments

To resolve this issue and allow your tests to run properly after upgrading to JDK 21, you need to add specific JVM arguments to your Gradle build configuration. Here's how to do it:

For Gradle-based Liferay Projects

Add the following to your build.gradle file in the test configuration section:

test {
    jvmArgs = ['--add-opens', 'java.base/java.lang=ALL-UNNAMED',
               '--add-opens', 'java.base/java.lang.invoke=ALL-UNNAMED']
}

For multi modules project you will need to add it to every module that uses Mockito.

My advice? Add this to all subprojects using top level build.gradle:

subprojects { subproject ->
    test {
        jvmArgs = ['--add-opens', 'java.base/java.lang=ALL-UNNAMED',
                   '--add-opens', 'java.base/java.lang.invoke=ALL-UNNAMED']
    }
}

This way you won't have to remember about that each time you use Mockito.

For Maven-based Liferay Projects

If you're using Maven instead of Gradle, these arguments in your pom.xml should do the trick:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>3.2.2</version>
    <configuration>
        <argLine>
            --add-opens java.base/java.lang=ALL-UNNAMED
            --add-opens java.base/java.lang.invoke=ALL-UNNAMED
        </argLine>
    </configuration>
</plugin>

How These Arguments Work

Let's understand what these arguments do:

--add-opens java.base/java.lang=ALL-UNNAMED: This opens the java.lang package from the java.base module to allow access from unnamed modules (which includes your application and its dependencies like Mockito).
--add-opens java.base/java.lang.invoke=ALL-UNNAMED: Similarly, this opens the java.lang.invoke package to unnamed modules.

These arguments effectively tell the JVM to relax its module encapsulation rules specifically for these packages, allowing Mockito to function correctly through reflection.

Is it new issue?

Well not really. I would say now it happens to many of us as Liferay has dropped support for JDK8 some time ago. Actually this is what happened to us when I first faced the issue: I was migrating the project to newest Liferay version and we were forced to update JDK as well. Earlier we did not want to do that due to other limitations we had and extra effort needed to resolve them.

Also just a month earlier someone I met on DevCon asked me on Slack about exactly the same specific error. This is why I thought I will create this as an article even though it is rather simpler article than the ones I normally try to publish.

Additional Considerations

Besides these "specific" error fixes you should of course also remember about updating your Mockito version to the newest one. Along with other libraries like Lombok which can also lead to unexpected tests errors for example with @Slf4j (which also happened to me updating Lombok did the trick so I'm not going into more details)

Conclusion

Upgrading to newer Liferay or JDK can bring numerous benefits but it almost always comes with some effort to make everything work. Nevertheless updates are important and should be done regularly which also simplifies each of them. Hopefully the solution above can help you at least a little bit to resolve Mockito related errors in your Liferay Portal projects after such upgrades.

Do you have any questions or doubts? Feel free to contact me directly.

Looking for expert Liferay development services? Contact InnRay for a consultation.

Liferay: Factory Configurations

Wed, 19 Jun 2024 00:00:00 GMT

Today, I faced an issue with so-called "Factory configurations" in Liferay, which are a way of defining custom configurations with multiple instances. I could not find any documentation or examples online, so I thought I would share the solution in this short article.

Custom configurations in Liferay

As you might or might not know, in Liferay, you can create custom configurations by defining a simple interface and using a few out-of-the-box annotations. What is cool about this is that the whole UI gets generated for you, so you do not have to deal with any portlets, JSPs, React portlets, etc. Another cool thing is that you can choose the scope of your configuration (system, instance aka company, site, portlet).

A basic example of such a configuration could be the following code:

package com.pydyniak.custom.configurations;

import aQute.bnd.annotation.metatype.Meta;
import com.liferay.portal.configuration.metatype.annotations.ExtendedObjectClassDefinition;

@Meta.OCD(
        id = "com.pydyniak.custom.configurations.CustomConfiguration",
        name = "My cool custom configuration"
)
@ExtendedObjectClassDefinition(
        category = "pydyniak",
        scope = ExtendedObjectClassDefinition.Scope.COMPANY
)
public interface CustomConfiguration {
    @Meta.AD(name = "My custom Config", description = "This is just an example configuration")
    String myCustomConfig();
}

which would result in the following configuration under Control Panel -> Instance Settings:

Custom Config

Using such a configuration is easy. We can just use the ConfigurationProvider OSGi service or ConfigurationProviderUtil, for example:

public CustomConfiguration getConfiguration(long companyId) throws ConfigurationException {
    return ConfigurationProviderUtil.getCompanyConfiguration(CustomConfiguration.class, companyId);
}

This is quite simple and straightforward. There are also a bunch of examples in the Liferay code (just look for the Configuration.java suffix to find them easily), and it is quite well described in the Liferay docs, so I won't go into more details.

Factory configurations

One thing that is kind of missing in configurations and is not widely used is the so-called "Factory Configuration". The factory configuration is a way of creating one configuration class, very similar to the one presented, but with extra options that make it possible to have multiple "instances" of this configuration. One example from Liferay is CORS settings, where you can define multiple entries:

Cors Settings

Doing that is really simple: in @Meta.OCD you have to use factory=true option and in @ExtendedObjectClassDefinition you need to define which field is used as a name in list view using factoryInstanceLabelAttribute = "CONFIGURATION_METHOD_NAME" where CONFIGURATION_METHOD_NAME is just a name of method (field) from your configuration. A full example could be a configuration where we want to configure multiple SSO partners with different settings (whatever they might be), and this was exactly the case I needed:

package com.pydyniak.custom.configurations;

import aQute.bnd.annotation.metatype.Meta;
import com.liferay.portal.configuration.metatype.annotations.ExtendedObjectClassDefinition;

@Meta.OCD(
        factory = true,
        id = "com.pydyniak.custom.configurations.SSOPartnerConfiguration",
        name = "SSO Partner Login Token"
)

@ExtendedObjectClassDefinition(
        category = "pydyniak",
        factoryInstanceLabelAttribute = "ssoPartnerName",
        scope = ExtendedObjectClassDefinition.Scope.COMPANY
)
public interface SSOPartnerConfiguration {

    @Meta.AD(required = false, name = "Enabled", deflt = "false",
            description = "Should SSO be enabled for this partner")
    boolean enabled();

    @Meta.AD(name = "SSO Partner Name")
    String ssoPartnerName();
}

The example above, of course simplified, shows how to create a factory configuration. In the portal, this would look like this:

SSO Partners configuration

and creating a new SSO partner:

Adding new SSO Partner

and once we create some SSO partners, we have:

List of SSO partners, after adding some

Doing that was quite easy. I could easily find examples in the Liferay code, such as the CORS configuration mentioned earlier: PortalCORSConfiguration, or the OpenID SSO configuration in OpenIdConnectProviderConfiguration.

What I could not find easily, though, was the way of accessing these configurations from code.

Accessing factory configurations in code

After some research I found a way of doing that - first of all we need to use ConfigurationAdmin service

@Reference
private ConfigurationAdmin configurationAdmin;

With this service, we can list configurations using the listConfigurations method. It takes a filter as a parameter, and what would it be? Well, looking at the sources, I found out that it has to have a format like: (&(service.factoryPid=CONFIGURATION_CLASS)(companyId=COMPANY_ID)) where:

CONFIGURATION_CLASS is just the class name of our configuration. In my example, it is: SSOPartnerConfiguration.class.getName()
COMPANY_ID is simply the company (instance) ID, as my configuration is company-scoped.

When we use that method we get array of org.osgi.service.cm.Configuration which is generally ok - we can get our properties out of it. We can go one step further, though, and find the configuration we need and convert it to our SSOPartnerConfiguration. To do that, we can combine the ConfigurableUtil#createConfigurable method with the Configuration#getProperties. A full example code would be:

@Reference
private ConfigurationAdmin configurationAdmin;

private Optional<SSOPartnerConfiguration> getSsoPartnerConfiguration(long companyId, String ssoPartner) throws IOException, InvalidSyntaxException {
    Configuration[] configurations = configurationAdmin.listConfigurations(String.format("(&(service.factoryPid=%s)(companyId=%s))", SSOPartnerConfiguration.class.getName(), companyId));
    return Arrays.stream(configurations).map(Configuration::getProperties)
            .map(properties -> ConfigurableUtil.createConfigurable(SSOPartnerConfiguration.class, properties))
            .filter(ssoPartnerConfig -> ssoPartnerConfig.ssoPartnerId().equals(ssoPartner))
            .findAny();
}

Summary

I hope this short article can help you with your development. It would definitely have saved me some time if I had found it in Google or in Liferay docs.

The configuration options are well described in Liferay docs, so I suggest you read through them if you are not familiar with the functionality. It's just this one piece of information that was missing; hopefully, this article is easy enough to follow.

As always, if you have any questions, do not hesitate to contact me.

Also, you can find related code on my Github.

Using Navigation Entry for custom functionality in Liferay core views

Mon, 30 Oct 2023 00:00:00 GMT

In some projects there is a need to override some functionalities provided by Liferay out of the box. There are multiple ways of achieving what we need and they are described in Liferay documentation. This extensibility is, in my humble opinion, one of the biggest assets of Liferay platform. Today I would like to share how can you avoid using OSGi fragments for JSP overrides in some cases by using custom Navigation Entries which can save you a lot of work during Liferay version updates.

Background

First lets describe some background and lets find an answer to following questions:

What are OSGi fragments
Why should you avoid using fragments for JSP overrides and why they are an issue while upgrading your Liferay version
Why even use OSGi fragments in real life projects

What are OSGi Fragments

OSGi fragment is a feature which can be used to extend or modify the existing functionality without modifying the code of original bundle. In Liferay context OSGi fragments can be used for example to override the complete JSP file from Liferay sources. Overriding JSPs is the most common use case for OSGi fragments in Liferay but not the only one. They can also be used for other things like exporting internal package which sometimes can be super handy.

Why should you avoid using fragments for JSP overrides and why they are an issue while upgrading your Liferay version

The overriding of JSPs is powerful but comes with cost. The main issue with this approach is that when you upgrade Liferay you basically have to verify every single overridden JSP and check if it is still working. In many cases it won't because:

Other JSPs which you didn't override changed and there are some dependencies between them
API changed: methods, or even classes, used in JSP do not exist anymore
Version mentioned in fragment host changed and your fragment does no longer attach to the overridden bundle

Moreover, you might easily miss that something is not working because JSPs generally do not produce compile errors and you can not rely on IDE as well as it will generally show a lot of errors in overridden JSPs anyway (because of dependencies which IDE does not know).

Even if you are not upgrading the fragments can sometimes be tricky and confusing, for example they can stop working without any errors on startup or during runtime. I even wrote why this could happen some time ago and how custom system checkers can help for such cases

Of course if you have one or two JSPs overrides it's fine. The issue is when you have dozens of them.

Why would you even use OSGi fragments for JSP overrides in real life projects

To override JSPs of course ;) This is actually really handy in many occasions where Liferay has some out of the box solution which only slightly does not fit to your use case. Some real examples from projects I've worked on:

Modification of login portlet to change redirection behaviour and add new fields/validations to registration form
Custom fields in addresses form and custom permissions handling
Adding custom field in Liferay Commerce Channel to describe what is the country of origin of channel for tax purposes

In fact this functionality can be used in many parts of Liferay code portlets. Both Liferay Portal and Liferay Commerce ones.

So what are navigation entries, the "heroes" of this post and screen navigation in general. Long story short - Screen Navigation framework allows you to create custom navigations for your application. In JSP it can be used with liferay-frontend:screen-navigation taglib. For examples you can of course look into the Liferay code where there are dozens of them. For example:

<liferay-frontend:screen-navigation
	context="<%= commerceCountriesDisplayContext.getCountry() %>"
	key="<%= CommerceCountryScreenNavigationConstants.SCREEN_NAVIGATION_KEY_COMMERCE_COUNTRY_GENERAL %>"
	portletURL="<%= currentURLObj %>"
/>

There is also a Liferay documentation describing screen navigation framework

So as mentioned the Screen Navigation is used in dozens of places in Liferay core code. We can take advantage of that and in use cases where we need to add new code (new fields to a form for example) we can define new Screen Navigation Entry instead of overriding the whole JSP.

This approach has some great advantages:

It is much easier to upgrade Liferay code - in most cases you will not need to do anything!
These are standard Liferay modules for which you will get normal runtime errors
Easier to find dependencies issues like using internal package, as there will be more verbose errors in logs

Going through the code

If you would like to use this approach you start exactly as you would like to find a JSP to override. My proposal is to check what Portlet is used on a page, then find it in sources and then figure out which JSPs you would like to override. Instead of overriding JSP with OSGi fragment though you will want to find out perhaps there is some navigation to which you could add your custom entry.

Let's take a real example. In one of the projects I've worked on there was a need to add a custom field to country. We needed to gather some extra information about each country. For the showcase purpose lets assume it is one field "EU country" which is needed to distinguish EU countries for tax purposes.

If you look into the portal (Commerce -> Countries) you will see that there is no way of defining that by default:

Country Details Page

So we need to find out which portlet is that. Just looking at the URL you can tell this is CommerceCountryPortlet. Next step would be to look into the Liferay sources where you can find all related JSP files:

Countries Portlet JSP files

In the screenshot above you can see three JSPs used for navigation in country details portlet:

Countries Details

Now we need to add new field. One approach would be to override the details.jsp with OSGi fragment with all the disadvantages mentioned before. Another approach is to see how is the navigation menu created and if we can extend it. In this case (any most of them) the Screen Navigation framework is used which we can see in edit_commerce_country.jsp

Screen Navigation in edit_commerce_country.jsp

Once we found a screen navigation we know that instead of overriding the whole JSP we can simply add new menu entry for additional fields we need.

The steps to do that are:

Create new Liferay module
Make sure to define Web-ContextPath in bnd.bnd file.
Add custom menu entry you will need two components: navigation category and navigation entry
Add JSP which should be rendered
Then of course we need to save our data somehow so we will need a MVC action command

Creating new module is something you probably know how to do but lets go through steps 2-5

Adding Web-ContextPath

This is really straightforward but if you have never done that before you might not know what to do here. Simply this is only one line which you need to add in bnd.bnd file of your module: Web-ContextPath: /country-custom-settings and it will define a ContextPath for the bundle rendering purposes. It is required as otherwise we won't be able to include our custom JSP.

Next step is to define two classes and mark them as OSGi components. First one is the implementation of ScreenNavigationCategory which defines a category in screen navigation. The class we need is really simple:

@Component(
        property = "screen.navigation.category.order:Integer=100",
        service = ScreenNavigationCategory.class
)
public class CountryCustomSettingsScreenNavigationCategory
        implements ScreenNavigationCategory {
    @Override
    public String getCategoryKey() {
        return "pydyniak-country-custom-settings";
    }

    @Override
    public String getLabel(Locale locale) {
        return LanguageUtil.get(locale, "custom-settings");
    }

    @Override
    public String getScreenNavigationKey() {
        return "commerce.country.general";
    }
}

Three methods we use here are:

getCategoryKey: used to define unique key for our category
getLabel: label used in navigation menu
getScreenNavigationKey: defines the key of screen navigation for which we want to add the entry. It has to be the same as the key defined in liferay-frontend:screen-navigation

Next we need an implementation of ScreenNavigationEntry. It has some more code which can be found in full version on my github

Here are the crucial parts:

@Component(
	property = "screen.navigation.entry.order:Integer=10",
	service = ScreenNavigationEntry.class
)
public class CountryCustomSettingsScreenNavigationEntry
	extends CountryCustomSettingsScreenNavigationCategory
	implements ScreenNavigationEntry<Country> {
    
	}

We are defining the navigation entry and marking it as OSGi component. The screen.navigation.entry.order:Integer can be used to define the order of navigation entries

@Override
	public void render(
			HttpServletRequest httpServletRequest,
			HttpServletResponse httpServletResponse)
		throws IOException {

		long countryId = ParamUtil.get(httpServletRequest, "countryId", 0L);
		Country country = countryService.fetchCountry(countryId);
		long companyId = portal.getCompanyId(httpServletRequest);
		setupEuCountryExpando(companyId);
		boolean euCountry = (boolean) country.getExpandoBridge().getAttribute(CountryCustomSettingsConstants.EU_COUNTRY_EXPANDO_KEY);

		httpServletRequest.setAttribute(CountryCustomSettingsConstants.PARAM_COUNTRY_ID, countryId);
		httpServletRequest.setAttribute(CountryCustomSettingsConstants.EU_COUNTRY_EXPANDO_KEY, euCountry);
		jspRenderer.renderJSP(servletContext, httpServletRequest, httpServletResponse, "/pydyniak/country_custom_settings_view.jsp");
	}

This method renders our custom JSP. Please note that I've used expando bridge framework to store the custom boolean value. The method "setupEuCountryExpando" source can be seen below:

private void setupEuCountryExpando(long companyId) {
		try {
			ExpandoBridge expandoBridge = expandoBridgeFactory.getExpandoBridge(companyId, Country.class.getName());
			if (!expandoBridge.hasAttribute(CountryCustomSettingsConstants.EU_COUNTRY_EXPANDO_KEY)) {
				expandoBridge.addAttribute(CountryCustomSettingsConstants.EU_COUNTRY_EXPANDO_KEY, ExpandoColumnConstants.BOOLEAN,
						false, false);
			}
		}catch (PortalException ex) {
			LOG.error("Error while setting up eu country expando", ex);
		}
	}

and it sets up the expando for Country. If it already exists then nothing changes.

Custom JSP

The render method from previous point mentions the JSP which is just a simple JSP (some parts skipped, see Github for full code):

<%@include file="init.jsp" %>

<%
    long countryId = (long) request.getAttribute(CountryCustomSettingsConstants.PARAM_COUNTRY_ID);
    boolean euCountry = (boolean) request.getAttribute(CountryCustomSettingsConstants.EU_COUNTRY_EXPANDO_KEY);
%>
<portlet:actionURL name="/commerce_country/edit_country_custom_settings"
                   var="editCountryCustomSettingsActionURL"/>

<aui:form action="<%= editCountryCustomSettingsActionURL %>" cssClass="container-fluid container-fluid-max-xl"
          method="post" name="fm">
    <div class="lfr-form-content">
        <div class="sheet">
            <div class="panel-group panel-group-flush">
                <aui:fieldset>
                    <aui:input name="redirect" type="hidden" value="<%= PortalUtil.getCurrentURL(request) %>"/>
                    <aui:input name="countryId" type="hidden" value="<%= countryId %>"/>
                    <aui:input checked='<%=euCountry %>'
                               inlineLabel="right" labelCssClass="simple-toggle-switch" type="toggle-switch"
                               name="<%=CountryCustomSettingsConstants.EU_COUNTRY_EXPANDO_KEY%>"/>

                    <aui:button-row>
                        <aui:button cssClass="btn-lg" type="submit"/>
                    </aui:button-row>
                </aui:fieldset>
            </div>
        </div>
    </div>
</aui:form>

The form is defined with only one field for user. Then on submit we call custom MVC action command

MVC Action Command

Once the form is submitted we want to call action url /commerce_country/edit_country_custom_settings. This MVC Action Command of course does not exist. We need to define one

@Component(immediate = true,
        property = {
                "javax.portlet.name=" + CommercePortletKeys.COMMERCE_COUNTRY,
                "mvc.command.name=/commerce_country/edit_country_custom_settings"
        },
        service = MVCActionCommand.class
)
public class EditCountryCustomSettingsMVCActionCommand extends BaseMVCActionCommand {
    @Reference
    private CountryService countryService;

    @Override
    protected void doProcessAction(
            ActionRequest actionRequest, ActionResponse actionResponse) {
        long countryId = ParamUtil.getLong(actionRequest, CountryCustomSettingsConstants.PARAM_COUNTRY_ID);
        boolean euCountry = ParamUtil.getBoolean(actionRequest, CountryCustomSettingsConstants.EU_COUNTRY_EXPANDO_KEY);
        Country country = countryService.fetchCountry(countryId);
        country.getExpandoBridge().setAttribute(CountryCustomSettingsConstants.EU_COUNTRY_EXPANDO_KEY, euCountry);
    }
}

We create a class extending BaseMVCActionCommand, then we define an action in which we set the value from the form as expando of Country object.

Results

If we deploy the module and open the country details again we will see new menu entry called "Custom Settings":

Custom Settings in Navigation

and the form:

Custom settings form

Note please note that I didn't mention all the code, especially the language module which provides the translation. For example can be found on GitHub in my Liferay Blog Samples repository

Summary

I hope you liked it and I encourage you to use this approach whenever you can instead of standard OSGi fragments. Sadly this approach is not always replacement, and it can only help us if we need to add new fields. It won't help in replacing existing ones and cannot be used to change things like order of fields, custom validations etc. For these you will still need the old approach.

Liferay: Why am I getting "Incorrect string value" error

Tue, 13 Dec 2022 00:00:00 GMT

Some time ago a colleague of mine struggled with an error "Incorrect string value: '\xE2\x86\x92)</...' for column 'description' at row 1" and asked me for a help. He tried to look into google and he found out that the error is connected to the database charset or collation settings but he couldn't find out why does it happen to him even though the settings seemed correct.

Let's find out why is that in this short article.

Database charset vs database collation

First we should understand what is the difference between these two to have better understanding of databases.

Charset is a set of characters that a database can store and manipulate. Each character in a charset is represented by a unique code point, which is a numerical value that corresponds to a specific character. The charset determines which characters can be stored and used in a database. Different charsets support different sets of characters, and they can be used to support different languages and writing systems. For example, the ASCII charset includes the basic letters, numbers, and symbols used in English, while the Unicode charset includes a much wider range of characters from many different languages and writing systems.

On the other hand, database collation is a set of rules that determine how the data in a database is sorted and compared. This includes rules for how character strings are ordered and how case, accent marks, and other language-specific features are treated. Different collation settings can be used to support different languages and regional conventions, and they can affect the way that queries and other operations are performed on a database. The difference between collations can be seen on example - letter Ł in polish is after L and before M and some collations will support that (such as utf8_unicode_ci) while others won't and will treat these two letters equally or they will just put the Ł after the Z. This could of course lead to wrong results of your queries.

As you can see these two topics are closely related, but they are not the same. Of course the way both are set depends on the particular database.

Examples of charsets and collations

Few examples of charsets and collations could be:

Charset: latin1, collation: latin1_swedish_ci - default charset/collation for MySQL up to version 5.7
Charset: utf8mb4, collation: utf8mb4_0900_ai_ci - default charset/collation for MySQL 8.0
Charset: utfmb4, collation: utf8mb4_unicode_ci

It's important to note that different databases have different default charset and collation settings, and these settings may need to be changed depending on the specific requirements of your application. Consult the documentation for your database to learn more about how to configure the charset and collation settings for your database.

Different "levels" for charset and collation settings

It's important to know that charset and collation can be set on different "levels". For example:

Server level
Database level
Table level
Column level

This of course can slightly differ for each database (MySQL, PostgreSQL, ...) so please check your database documentation for more details. Anyway what is important for us at this stage is that even though your database can have a proper settings the specific column can have wrong settings. How can this happen? Well for example if following situation happened:

Schema was created with wrong charset/collation
Liferay was started
Someone fixed the schema settings

When Liferay starts it creates a lot of tables. By default, it uses the database level charset/collation settings for all the tables. Changing the database settings won't fix the wrong settings for specific columns. And this is a possible case when database seems to have correct settings but Liferay still doesn't start up properly.

Solution to "Incorrect string value" issue

If you read the explanation I bet the solution might already be clear for you - you just have to make sure that:

Your database has correct charset and collation settings
Your existing tables have correct charset and collation settings

Please note it's not enough to just fix already created tables as the new ones could still get the default, wrong settings. That's why you always it's better to always change settings on both levels.

Consult the documentation for your database to learn how to properly configure the charset and collation settings. For MySQL examples see commands below.

I will only describe MySQL commands here as an example but you should be easily transfer them to your database or find a solution online.

Checking DB default settings

SELECT * FROM information_schema.SCHEMATA
WHERE schema_name = 'liferay';

"Renaming existing columns"

Checking specific table settings

SELECT T.table_name, CCSA.character_set_name as column_character_set,
       CCSA.COLLATION_NAME as collation_name
       FROM information_schema.`TABLES` T, information_schema.`COLLATION_CHARACTER_SET_APPLICABILITY` CCSA
WHERE CCSA.collation_name = T.table_collation
  AND T.table_schema = 'liferay'
AND T.TABLE_NAME='AccountEntryOrganizationRel';

"Renaming existing columns"

Changing all the tables settings

It's a little bit more complicated - there is no out of the box command to change settings for all the tables at once. It can be done with a little workaround though.

SELECT
    CONCAT('ALTER TABLE ', T.TABLE_SCHEMA, '.', T.TABLE_NAME, ' CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;') AS SQL_UPDATE,
    T.table_name, CCSA.character_set_name as column_character_set,
    CCSA.COLLATION_NAME as collation_name
FROM information_schema.`TABLES` T, information_schema.`COLLATION_CHARACTER_SET_APPLICABILITY` CCSA
WHERE CCSA.collation_name = T.table_collation
  AND T.table_schema = 'liferay'
  AND CCSA.CHARACTER_SET_NAME='latin1';

Renaming existing columns

And then in first column you have a list of ALTER TABLE which you have to run. Just copy & paste the list and run it. Each table collation will be changed one by one.

Final words

As you can see this error is not too complicated. It's good to have the basic understanding of charsets and collations though that's why I tried to explain it a little bit as well.

I hope you liked it and found it useful.

Is Liferay Commerce production ready? Personal Experience

Mon, 03 Oct 2022 00:00:00 GMT

Some time ago Liferay Commerce has been introduced. First it has been shipped as a separate product. Then it got included in Liferay and no separate installation was required. For new versions of Liferay you do not even have to turn it on as it is turned on by default (you can still turn it off if you want).

If you use Liferay then you have probably already came across this new functionality but have you ever wondered - is this really useful and production ready?

I will try to answer this question in this post based on my personal experience.

Few words about my experience with Liferay Commerce and why did I even create this post?

You might wonder what kind of experience do I have with Liferay Commerce and why did I create this post in the first place. Let's start with the experience - I have worked with Liferay Commerce for about a year. During this year I've used probably every aspect of it - from the basics like channels, catalogs to things more detailed like price lists, relations of articles, detailed settings of shipping options and more. Not only I have used out of the box functionalities, but I also extended/overrode different parts of the system and looked into tons of different options to see how we can use what is already there for our particular needs (or to be more precise - for our customers needs). Especially with multiple interfaces with ERP systems we needed to support.

With all of that experience I feel like now I know it quite well, or at least fairly well given it is still quite a new product.

Because this is a new product and there are not too many people who could share their success (or failure) stories I thought I will at least share some of my thoughts. This way some of you who already use Liferay might be in a better position to decide if you should also use it for your company's shop site.

Pros and cons of Liferay Commerce

In my opinion Liferay based on their previous experience (including mistakes) has learned a lot, and they tend to do new functionalities much better, stable and easier to extend. This also includes the commerce connected modules - in general they work surprisingly good but of course there are also some downsides.

I will go through its few pros and cons, so you have better feeling what to expect from it.

Pros: Variety of options

To be honest I haven't worked with different shop engines like PrestaShop, WooCommerce, Magento etc. This means I can't fully compare the functionality between them and Liferay Commerce. I looked briefly on documentation of some of them but of course this doesn't make me an expert.

What I can tell you though is that the shopping configuration possibilities in Liferay Commerce are really impressive. Moreover, you can use the possibility to have different settings per instance / per site and, in commerce context, per channel. This means you can host one site meant only for B2C, another one only for B2B and third one could of course be B2X. You can have different shipping/payment/tax options for each of them. You can have different price lists applied for different customer accounts or accounts groups. You can have multiple catalogs which could be shared on different channels. You can have different flows, price types (net vs gross), different relations between products, payment/shipping methods and restrictions, different notification (emails) templates and many more.

I won't of course describe everything as it's not the point here. What I would like to say though is that I personally used many of these options and indeed the out-of-the-box functionalities seems to be production ready. Of course, you can still have specific needs. This is a standard thing for big projects. The good thing is that extensions options are huge and quite simple to do.

Pros: Documentation

Many developers I've worked with have the feeling that documentation has never been a strong side of Liferay. Luckily, at least in my opinion, it has changed quite significantly recently. With new, revamped documentation I feel it gets much better. Regarding Liferay Commerce the documentation describes not only basic stuff but also shows examples of how to implement things which are less popular like custom tax engine , new payment method and more.

Pros: Extending options

Like I mentioned before - the documentation describes quite a lot of options of extending Liferay Commerce. If you have worked with portal before and if you have extended it you probably already know that there are variety of options for overrides/extensions: fragments, portlet filters, custom implementations of services leveraging OSGI etc.

All of these things still exist, and you can use your knowledge with Commerce modules as well. Moreover, the design of newly developed modules is, in most cases, created in a way that creating the overrides are even simpler than before. For example many tables use the Clay datasets which makes it easier to change tiny things - you don't have to touch the JSP view but instead you can just override Java components. It's quite convenient in many cases.

Having said all of that, be prepared that you will still have to take a look on the Liferay sources to find extensions points.

Nevertheless, I would say that possibilities of extensions and ease of doing them is one of the biggest advantages of Liferay. Both for the standard CMS and the Commerce modules.

Pros: Liferay huge ecosystem

If we talk about the biggest advantages of Liferay another thing that comes to my mind is huge ecosystem of Liferay itself. Implementing a shop is one thing but there is a lot Liferay can offer beside that.

You need regular content pages? No problem. Just use standard CMS functionalities
You need advanced workflow for creating content? You've got it.
Advanced users/organizations/permissions options? Of course!
Multitenancy?

Well I even wrote an article about that.

Integration with LDAP for synchronizing the users? Or perhaps SSO made with Kerberos or SAML? Sure!

Basically everything I wrote about advantages of Liferay and many other things I haven't (because there are so many) applies to Commerce as well as, after all, it's the same product.

This is huge advantage. Especially if you're providing portal (/shop) to multiple customers/suppliers and everyone has a slightly different requirements.

Cons: Some issues / poor design still happens

While they fix issues quite regularly and that's a definitely a pro, there are still some things which were not designed too well. Example? Here is one but it requires a little introduction.

Generally in Liferay there are some built-in shipment methods. For now these are FedEx, Flat Rate and Variable Rate.

Out of the box shipment methods

The variable rate and flat rate can contain some additional options. Meaning the method is variable rate but the particular option can be anything you want - for example you can have: "Self pickup", "Carrier", "Carrier express". For each option you can set up costs for this option based on the weight of the order, destination and warehouse. This is really great out of the box functionality. The issue? Well in the early versions of 7.4 the shipping option picked for order hasn't had any key or ID reference. Only the shipment option name was saved in the database, and it was saved for language in which the checkout has been made... Problem? Well name of the option can change. It's a totally configurable thing, so you never should depend on it.

This thing has been fixed but if you are like me and have a huge project which you don't upgrade every month or two, because of costs and time restrictions, then you have to stick with that.

Of course, it can happen in any product, but I believe it should not happen in such core functionality especially that it is not some issue introduced by a mistake but has been in Liferay Commerce for a long time.

Cons: Responsiveness..

This might be surprising but Liferay still is not fully mobile friendly. Of course, I think no one would expect that they make the whole administration panel responsive but at least the things meant for end-user should be responsive. Sadly they are not. Even the Liferay Commerce which is kind of new product has some issues.

Examples? Here are some:

Products list in checkout summary step
Accounts management - list of accounts
Placed orders list

This is how placed orders look like on Iphone 12 Pro

Basically Liferay still has an issue with proper, responsive display of lists. What we get is simple horizontal scroll for the list. That's a shame in my opinion as there are better solutions. You might be able to make it look better with only styling in theme but some solutions might also require changing the HTML structure which would require using JSP fragment, which is not exactly recommended as it makes the further Liferay upgrades harder.

Of course for some projects it might be enough. I've spent a lot of time for lists display changes though.

Just to make it clear - I even tried the newest Liferay version (7.4.3.44-ga44) to check if maybe it looks better now, but it's not.

Cons: Community

Liferay is not the most popular CMS nor shopping engine. There are a bunch of companies using it and being happy about it but the fact is that its market share is not too high. Or to be more precise it is quite low. This also means that the community is rather small. Often you might not get the help you need, or you will get it after long time. This means more time spend on debugging of the core code.

I feel like this is getting a little better though - especially on Liferay's official Slack which I highly recommend. From my experience questions asked there gets much more answers than the ones asked on the forum. Still I can see that many questions, mostly the harder ones, gets unanswered. With more popular systems there are more people experiencing the same issues and higher chance to get an answer.

Cons: It's still Liferay - most likely not for small business

Like I have already mentioned in one of my previous articles about pros and cons of using Liferay I think Liferay in general is not for small business. With Liferay Commerce it's the same story.

Why? Mostly because of small community (see previous point) which makes it hard to find answers for your questions. Another thing is that you won't find too many plugins so if you can't make something using out of the box functionality and you are not developer yourself then you will be forced to hire someone to extend the portal for you. Of course that's the case with any other solution but the thing with Liferay is that it is not too common which also makes it harder to find employees or contractors who know it. This means slightly higher rates as well. Of course, it still can make sense to use Liferay & Liferay Commerce! It's just probably not what you want if you're just a tiny business with two or three employees and your budget is not too high.

Summary

In my opinion usage of Liferay Commerce makes sense. In some cases it might be actually a great solution. Having said that, in small projects I would not decide to use it, unless you already know Liferay or have Liferay team. Of course after all it all depends on your specific needs. I just wanted to give you some overview which I hope can help you to make a decision if you should even take it for consideration.

If you have some more questions feel free to contact me at rafal@pydyniak.pl or through my LinkedIn.

Using Custom System Checkers for finding unresolved bundles

Fri, 22 Jul 2022 00:00:00 GMT

In this post I would like to explain what System Checkers in Liferay are, how you can implement one and what you can achieve using it. As an example I will use a real life scenario - what if you have a fragment which depends on another OSGi module and this module stops exporting the package you need in your fragment. In such scenario Liferay will sadly not report any error even though your fragment basically stops working. Lets find out how we can use system checkers to find such situations or how you can use them for any other checks you need.

Some introduction

So the situation I described at the beginning is the real issue I've experienced. In a project I work on we have quite a lot of different fragments. Some of these fragments make some huge changes in the basic views while others do only a tiny changes like adding a field, button or some text. Sometimes these fragments use constants from another API modules.

In my blog samples project I created such fragment as an example. This fragment is meant to add a simple text in the "Password" page of editing user view (Under Control Panel -> Users and Organizations -> Edit one of the users).

This fragment uses the API module for a constant text (lets skip translations in this example). The added code looks like this:

<p>
    <%=FakeConstants.FAKE_CONSTANT%>
</p>

Nothing really fancy here if you are familiar with fragments modules.

Anyway the issue with this fragment is that the FakeConstants class package is not exported by the API module's bnd.bnd file. Because of this the fragment will not start. It will stay in the Installed OSGi state

But hey - developer would notice that the fragment has not started

Of course. He would. And even if he pushed that without testing someone else would. For example tester or project manager or even a client. That's true.

But what if the module has been created few months ago, everything was working but then someone moved our class to another package because he thought it's only used internally in the same module? Or even removed the Export-Package line from bnd.bnd?

Well in such scenario the edit password page, which has worked for long time, might not be retested and the issue could've been overseen. Of course the page would still work but without changes from our fragment module. It would just use the default Liferay's view. This can be fine in some cases but maybe in some cases it's a legal requirement which your fragments fulfills for example adds some specific warning text required in your country. Or maybe your fragment adds some extra fields to the form and another module you use needs these fields otherwise the form won't work.

Doesn't Liferay give you some error on startup if the bundle couldn't be resolved?

Well actually it does but only for normal modules. In order to show this I created another module: not-resolved-bundles-web which is a basic portlet and it also uses the same, not exported API. In this case on startup there are errors in the logs:

2022-07-22 07:32:59.859 ERROR [fileinstall-directory-watcher][DirectoryWatcher:1173] Unable to start bundle: file:/opt/liferay/osgi/modules/com.pydyniak.blog.samples.not.resolved.bundles.web.jar
2022-07-22T07:32:59.860166363Z com.liferay.portal.kernel.log.LogSanitizerException: org.osgi.framework.BundleException: Could not resolve module: com.pydyniak.blog.samples.not.resolved.bundles.web [1503]_  Unresolved requirement: Import-Package: com.pydyniak.blog.samples.not.resolved.bundles.api_ [Sanitized]
2022-07-22T07:32:59.860172750Z 	at org.eclipse.osgi.container.Module.start(Module.java:444) ~[org.eclipse.osgi.jar:?]
2022-07-22T07:32:59.860182947Z 	at org.eclipse.osgi.internal.framework.EquinoxBundle.start(EquinoxBundle.java:428) ~[org.eclipse.osgi.jar:?]
2022-07-22T07:32:59.860184568Z 	at com.liferay.portal.file.install.internal.DirectoryWatcher._startBundle(DirectoryWatcher.java:1156) [bundleFile:?]
2022-07-22T07:32:59.860186144Z 	at com.liferay.portal.file.install.internal.DirectoryWatcher._startBundles(DirectoryWatcher.java:1189) [bundleFile:?]
2022-07-22T07:32:59.860187736Z 	at com.liferay.portal.file.install.internal.DirectoryWatcher._process(DirectoryWatcher.java:1046) [bundleFile:?]
2022-07-22T07:32:59.860189692Z 	at com.liferay.portal.file.install.internal.DirectoryWatcher.run(DirectoryWatcher.java:221) [bundleFile:?]

But the fragment doesn't give you the same errors. In fact Liferay doesn't give you any message in such case.

The issue is that the DirectoryWatcher which is responsible for starting modules works differently for fragments and for other modules. The fragments modules as you might now cannot be started, they can only be resolved (meaning all dependencies have been met) and then they works.

Then how can we find out that the fragment has not resolved?

Of course we still can find out that the fragment has not been resolved by using gogo shell. We can use the lb command which lists all the bundle. Then we can find bundles which are in "Installed" state and not resolved or active. Using this method we can find our two bundles:

Our two bundles in installed state

Tip In the Gogo Shell we can also use grep to filter the results:

lb | grep "Installed"

Then we can use the diag command to find out why particular bundle hasn't started:

Example of using diag command

The main issue I have with that approach is that you have to manully use lb. In fact there is even an OSGi command called system:check which is supposed to find all the issues in the deployed bundles but it just doesn't show you any issues in our scenario:

No errors using system:check command

And this is what we're going to change

System Checkers

What system checkers are

System checkers are the components which do different checks of your OSGi bundles. The idea is that you can run system:check and find out if everything is fine.

Also the system check is run automatically on every startup if you have a following portal properties entry:

module.framework.properties.initial.system.check.enabled=true

It works quite well for normal modules but like shown before you can't fully rely on the default system checkers if you're using fragments.

How we can implement a custom checker - example of creating one which finds installed (but not resolved) bundles

So we know that the default system checkers are not reliable in all situations but we can fix that with custom system checker which we will create on our own. The idea is really simple. We have to:

Create a class implementing SystemChecker interface
Register our class as a component
Implement all methods from the interface

So lets start with an empty class:

public class UnresolvedBundlesSystemChecker implements SystemChecker {

}

Then add a Component annotation:

@Component(
    immediate = true,
    property = {},
    service = SystemChecker.class
)
public class UnresolvedBundlesSystemChecker implements SystemChecker {

}

This interface has three methods we have to implement

public interface SystemChecker {

    public String check();

    public String getName();

    public String getOSGiCommand();

}

Lets start with the second and third ones. The getName one is supposed to just return name of the checker:

@Override
public String getName() {
    return "Unresolved Requirement System Checker";
}

The getOSGiCommand should return a OSGi command which can be run to execute this system check on its own. This command name is later logged in the system:check result.

@Override
public String getOSGiCommand() {
    return "pydyniak:unresolved";
}

You can name the command as you want. I will show you the implementation of the command component later.

Now lets go to the check method. This is the method that is being called when you run the system:check and it's the one where you should implement all your logic. In the end you have to return a String which will be displayed by system:check but of course you can add some extra logic there for example use Log to also log the issues to the log files.

How the logic should look like in our example? Well lets start with the fact that you can get information about bundles in OSGi by using BundleContext which you can obtain in the activate method of your component. So lets get the BundleContext and save it:

private BundleContext bundleContext;

@Activate
protected void activate(BundleContext bundleContext) {
    this.bundleContext = bundleContext;
}

Once we have that we can use that in our check method. I will first create an Util class which will handle our logic:

public class UnresolvedBundlesUtil {
    private UnresolvedBundlesUtil(){}

    public static String listUnresolvedBundles(BundleContext bundleContext) {
        return Arrays.stream(bundleContext.getBundles())
                .filter(UnresolvedBundlesUtil::isInInstalledState)
                .map(UnresolvedBundlesUtil::getMessage)
                .collect(Collectors.joining());
    }

    /**
     * We only look for installed bundles. This is the states in which bundles might have unresolved requirements
     * @param bundle
     * @return
     */
    private static boolean isInInstalledState(Bundle bundle) {
        return Bundle.INSTALLED == bundle.getState();
    }

    private static String getMessage(Bundle bundle) {
        StringBuilder message = new StringBuilder();
        message.append("\n\tBundle {id: ");
        message.append(bundle.getBundleId());
        message.append(", symbolicName: ");
        message.append(bundle.getSymbolicName());
        message.append("} is unresolved");
        return message.toString();
    }
}

So what are we doing here - the listUnresolvedBundles method take BundleContext object as a parameter. Then it uses it to obtain all the bundles in your OSGi container. In our example we want to find bundles that are not resolved. In OSGi world the bundle which couldn't be resolved is a bundle for which not all dependencies have been met. In such scenario it stays in Installed state. When new bundles are deployed our installed bundles are checked again in order to verify if the requirements have been met or not. If yes then the installed bundle gets resolved as well.

Note if you're not familiar with OSGi states you can read some more about them for example in JavaDocs of Bundle or in Liferay docs (they have some nice pictures) or in many different places across the Internet.

Checking the bundle state is quite easy and can be seen in isInInstalledState method:

private static boolean isInInstalledState(Bundle bundle) {
    return Bundle.INSTALLED == bundle.getState();
}

Then once we find bundles that are installed we can create some basic message for each of them:

private static String getMessage(Bundle bundle) {
    StringBuilder message = new StringBuilder();
    message.append("\n\tBundle {id: ");
    message.append(bundle.getBundleId());
    message.append(", symbolicName: ");
    message.append(bundle.getSymbolicName());
    message.append("} is unresolved");
    return message.toString();
}

Both methods mentioned above are used in the listUnresolvedBundles which uses Lambda to do all the work:

public static String listUnresolvedBundles(BundleContext bundleContext) {
    return Arrays.stream(bundleContext.getBundles())
        .filter(UnresolvedBundlesUtil::isInInstalledState)
        .map(UnresolvedBundlesUtil::getMessage)
        .collect(Collectors.joining());
}

Going back to our check method in UnresolvedBundlesSystemChecker class - we simply call our listUnresolvedBundles method and return the result:

@Override
public String check() {
    return UnresolvedBundlesUtil.listUnresolvedBundles(bundleContext);
}

Creating a OSGi command

In the beginning we defined custom OSGi command called pydyniak:unresolved which can be executed to run the system checker on its own. Sadly this won't call our check method by default. Luckily we can easily create the OSGi command on our own. To do that we have to create a simple component which again calls our UnresolvedBundlesUtil class and then simply uses System.out.println to log the result in the gogo shell:

@Component(
    immediate = true,
    property = {"osgi.command.function=unresolved", "osgi.command.scope=pydyniak"},
    service = UnresolvedBundlesOSGICommand.class
)
public class UnresolvedBundlesOSGICommand {
    private BundleContext bundleContext;

    @Activate
    protected void activate(BundleContext bundleContext) {
        this.bundleContext = bundleContext;
    }


    public void unresolved() {
        System.out.println(UnresolvedBundlesUtil.listUnresolvedBundles(bundleContext));
    }
}

If you have created OSGi commands before then it should be simple. If not then it's enough to understand that the osgi.command.function and osgi.command.scope define how our command can be run and then we just need a function named exactly the same as the osgi.command.function in our case it's unresolved.

Testing

Once we have everything we can get to the testing. First we must deploy our modules. Then we should go to the Control Panel -> Gogo Shell and execute system:check command (it takes a while). In the result we can see our custom system checker being run and its result:

Our custom system checker results

We can also test our custom OSGi command, which is mentioned in the system:check result:

Testing the pydyniak:unresolved command

Now we can be sure that our code works just fine.

Final thoughts

With this quite simple module we can be more certain that if we run system:check we get all information about the errors.

As far as I'm concerned there are no other possible errors in bundles which could not be found with system:check (but I might be wrong ;)).

System checkers of course could be use for different things, even not related to the bundles or components states. Same for OSGi command - it's really cool feature in my opinion which I have used in few occasions and I really like them. They're quite well documented so if you have any doubts you can read the Liferay documentation.

On the other hand, the System Checkers are not documented at all, at least not for the moment. You can only find information about how to run the system check but not how to implement custom ones. That's why I decided to create this blog post.

The whole example (API, Service, Custom Portlet and Fragment) can be found on my Github: https://github.com/RafalPydyniak/liferay-blog-samples/tree/master/modules/not-resolved-bundles

Of course feel free to use the code in any way you want.

Like always I hope you learned something and of course if you have any questions just email me.

Overriding Liferay Core Permissions

Mon, 23 May 2022 00:00:00 GMT

In Liferay you can define custom permissions for your entities/portlets. For example you can say that in your application in the employees management panel there are four actions: "List employees", "Add employee", "Edit employee", "Remove employee". Roles can then be associated with these permissions as required. Definining that is quite simple and described in Liferay docs which you should read if you haven't done that before.

But what if you need to override some permissions which were defined in Liferay core modules? Well you can of course do that as well. If you're wondering how - keep reading.

Note there are some articles on my blog which do not require any specific knowledge. This article is not one of them. I'm trying to make it as simple as possible so you don't need to be a Liferay expert but some basic knowledge about Liferay or OSGi might be required to follow.

Note 2 you can find a complete sources for article on new github repository I created for Liferay code samples

Why would you want to override Liferay core permissions?

There might be a different answers to the question in this section header. I have encountered different reasons to override Liferay permissions in different projects but in general I think there are two main reasons to do that:

You're not happy with default permissions given to Guest or Site member roles.

If you're already familiar with defining permissions in Liferay you probably know that you can define what permissions are given to guest/site member by default. Liferay core permissions also do that. For example by default GUEST has VIEW permission on every JournalArticle: What if we don't want that? If we want to control it a little bit more and by default don't give anyone a VIEW permission on any article? Well in such scenario you might want to override Liferay core to remove this part.

Second reason might be that you want to add some custom action for some Liferay core portlet or model permissions.

You might for example want to add a custom permission to check if user can checkout an order.

Things to consider when overriding permissions - when you should not do it?

Even though overriding the permissions is quite simple which you will see in a few moments there are things you need to consider:

In most cases you should not remove actions or if you really want to do that - be really careful with that

Of course, in general you could do that but keep in mind that overriding the default.xml (or any other *.xml with permissions definition) only defines what permissions are available. All the permissions checking on the other hand is done in the Java/JSP code. What does it mean for you? Well if you remove some resource action you might end up with errors connected to checking permissions which (after your changes) do not exist. If you would really want to remove some permission you need to make sure to remove all calls to it as well which means more Liferay core overrides.

Adding permission does not mean checking for permissions. In permissions *.xml files you only define what is available in Liferay

All the checking if user has the permission is done elsewhere (Java/JSP code). So even if you add some custom permission to some core portlet or entity you still need to add a logic of checking it. So in general adding custom actions makes sense mostly if you also hook the module in which you add some extra block of code and where you can add usage of your permission.

OSGi vs non OSGi modules - two different approaches for overriding permissions

To start the work on permissions override you first need to determine where the permissions are defined. How to do that? Well you should have a local copy of Liferay portal (available on Github) for the version your using (checkout to a specific tag for your version).

The next step is to find a correct module in the code. Finding code you need in Liferay core is a topic for a separate article - there are many ways to do that.

For finding the definition of permissions it's luckily a little easier. First go to the "Define permissions" section of any role. Then find the block of permissions where you want to change something. For example in my case I want to change the "Web Content Article" default permissions:

Web Content Article permissions

Once I'm there I can go to the browser's developer tools and see the id of title element:

Title of permissions

We can see it is "resource_com_liferay_journal_model_JournalArticle" and we're interested in the last part. The "JournalArticle" is the model name we're interested in. In this case you could probably guess the name if you work with Liferay for some time - JournalArticle is a well known name. In some other cases though this might let you find what you need.

Next step is to search for this model name in Liferay sources - you will get hundreds of results - this is too much. Luckily we can filter that:

Use the file mask to filter only "*.xml" files - the permissions are defined in XML files so we can use that.
Use the <model-name> tag with the name you found earlier and a little regex.

These two filters should be enough to find the xml file you need:

Find permissions definition XML

In this case only three files were found. Thirty seconds later you should be able to find out that the file we're looking for is the "journal.xml" which is located in journal-service module. Looking at the module structure you can tell that this is an OSGi module.

Override definition of permissions in OSGi module

Once you know your module is OSGi one you can start work on overriding permissions. The idea is simple - you need a fragment which will override permissions XML files. Just like you would normally override JSP file.

These are the steps you need to take:

Create a fragment module.
Specify a Fragment-Host property in bnd.bnd.
Create resources structure and override XML files.
Deploy your module.

Create a fragment module

You can crate a fragment using blade create command or your IDE. You can also take a look on a module I already created in blog samples.

Specify a Fragment-Host

Once you have your module you want to make sure that your fragment points to a proper bundle version. You can do that in bnd.bnd file by using Fragment-Host property. In my example I will override permissions from journal-service module. Checking the Liferay sources I can tell that I need to override the com.liferay.journal.service (see the bundle symbolic name in bnd.bnd file of overridden module). You also need to check the bundle version - either in sources or in gogo shell:

Checking bundle version in gogo shell

Once you have bundle symbolic name and its version you can define Fragment-Host in your module bnd.bnd file:

Checking bundle version in gogo shell

Creating resources structure and override XML files

Once our fragment points to a proper Liferay's bundle we can start overriding the permissions files.

We need following files/directories:

In the resources directory we need to create a resource-actions directory where we will put updated versions of

XML files we want to override.

In my case I want to do the changes in "journal.xml" file. To do that I copy the journal.xml from Liferay sources

with changed name - for example "journal-ext.xml".

You also need to copy the "default.xml" and also change its name

for example to default-ext.xml.

As we will create some new resource actions we can also create a "content" directory

where we will keep our translations.

The last thing you need to do is to create portlet.properties file

in resources directory.

In the end you should end up with following structure:

Files structure for journal.xml override

Now lets start with filling all the files.

First portlet.properties. In this file we want to specify that our default-ext.xml file should be used instead of the original default.xml:

include-and-override=portlet-ext.properties
resource.actions.configs=resource-actions/default-ext.xml

In the default-ext.xml itself we want to change only one line from the original file content - like you might guess we want to use our journal-ext.xml instead of the original journal.xml. So the original default.xml file content is:

<resource-action-mapping>
    <resource file="resource-actions/journal.xml" />
    <resource file="resource-actions/journal_ddm_composite_models.xml" />
</resource-action-mapping>

And our override should look like:

<resource-action-mapping>
    <resource file="resource-actions/journal-ext.xml" />
    <resource file="resource-actions/journal_ddm_composite_models.xml" />
</resource-action-mapping>

Note We don't need to copy other files like journal_ddm_composite_models.xml. If we keep the original name then the original files will be used.

All the changes above are required to finally make our updates in the journal-ext.xml (which is the copy of original journal.xml!).

You can do whatever you need there. If you're not familiar with defining permissions you should check the original Liferay documentation. In my case, for showcase purpose, I will do a few changes.

First I will add an extra "TEST" action permission for JournalPortlet:

Extra "TEST" permission for Journal Portlet

I will also remove the default ADD_DISCUSSION and VIEW permissions for guest:

Removed permissions for Journal Portlet

Since we added a new action, which has not been yet translated, we should also create a translation for it. That's where we need translation files which we created earlier. In my case I only have a single "Language.properties" but of course you can add different translations. In these files you can add translation like you would normally do in any other module. Just keep in mind your translation for resource action has to start with action. prefix:

action.TEST=Test permission

Deploy module

The deployment in that case is simple and "standard". Simply build a JAR & deploy it to the deploy directory of your Liferay.

Testing our changes

That's it! Once deployed your permissions are ready:

Test permission visible in roles panel

Override definition of permissions in non OSGi module

Sometimes you also have to deal with overriding permissions in modules which are not OSGi ones.

Note I think, at least for now, the portal-impl is the only non osgi module where permissions are defined

Note 2 The steps I describe worked for me for different 7.4 versions (including GA18 which is used in the blog samples) and I think it will work in future as well but you can never be sure. This isn't a standard way of handling things in Liferay. Sadly I haven't found any better way of doing that.

This means you can't simply use fragments to override them. Luckily we still can make it work.

The steps are quite similar to the ones we did for OSGi one so for the same things I will not describe them again. Basically we need to (I'm overriding documentlibrary.xml from portal-impl but the steps are of course the same for other XML permissions files):

Create a module.
Create default-ext.xml and documentlibrary-ext.xml (in my example! You might need another one) files - just like we did

in OSGi module override.

Add our default-ext.xml to the portal-ext.properties or to the Liferay environment variables for Docker environments.
Deploy our module to proper place.

Like you can see it's almost the same with tiny changes - we don't need to define portlet.properties and we don't need to define a Fragment-Host in our bnd.bnd. This is because we're not using fragments here.

Inform Liferay about our custom default-ext.xml file

After you created your module and made your changes in default-ext.xml and documentlibrary-ext.xml, which I won't describe again but you can take a look on a sources , you need to inform Liferay about our custom default-ext.xml file You can do that by adding a line to your:

portal-ext.properties
Docker environment variables

depending on your environment type.

For the portal-ext.properties file it should look like that:

resource.actions.configs=META-INF/resource-actions/default.xml,resource-actions/default.xml,resource-actions/default-ext.xml

While for docker environments the env variable should look like:

LIFERAY_RESOURCE_PERIOD_ACTIONS_PERIOD_CONFIGS=META-INF/resource-actions/default.xml,resource-actions/default.xml,resource-actions/default-ext.xml

What exactly is the value we used? Well this property defines list of resource action configurations which are read from classpath. So you first have to check up the original value in portal.properties file of Liferay's portal-impl module

Original resource-actions value

and then add your default-xml. Of course, you can name it differently.

Deploying our module

Now how do we deploy that? We can't simply deploy our OSGi module as usual - it won't work. Instead, we need it to be available in classpath during Liferay start up. To do that we should place our JAR into tomcat/lib directory.

So basically:

Create a JAR
Put it into liferay/tomcat/lib/ directory
Restart the server

In my liferay-blog-samples I also overrode the Gradle's deploy task to make it more automatic

deploy() {
into "$rootDir/docker/liferay/tomcat/lib"
}

But of course keep in mind this works because I use Gradle and the samples have the docker-compose environment defined in the same repository.

Testing our changes

Once we do that we can verify that our changes work:

Extra XYZ permission for document

As you might've seen I haven't specified any translation for my XYZ resource action. Of course, you can do that just like in the OSGi example.

Possible issue with overriding portal-impl permissions on first Liferay startup

One issue I have encountered while overriding portal-impl permissions definitions is that it won't work if you start up Liferay for the first time, with empty database when Liferay creates tables and basic data.

So instead you should :

Let Liferay start for the first time.
Deploy your portal-impl override and add portal-ext/docker env property.
Restart the server.

Of course this won't be necessary in existing projects where you already have some data in a database.

I don't know why this happens but since this is only an issue if you're starting a totally fresh database I didn't go into the details and didn't try to find exact reason.

Final thoughts

I believe changing permissions in Liferay core modules should be quite easy for you know. The ways of doing that are not that hard - especially for OSGi modules it's easy and straightforward but at the same time these things are not too well documented and in some projects I have seen some weird ways of doing such changes. That's why I decided to create a small blog post to show how it can be done in quite easy way.

Of course maybe there is still some place for improvement - if you know some, let me know at rafal@pydyniak.pl and I will update this article.

Good luck and if you need any help you can always send me an email. I will do my best to help.

Achieving multitenancy using Liferay's Virtual Instances

Fri, 16 Jul 2021 00:00:00 GMT

Imagine you create an app that you will sell to many different companies. It can be for example some application for managing company's finances or an app that will simplify holidays requests flow in a company. What is important is that you will sell that to different entities (companies, universities or whatever) and one client should not know anything about other clients that use your application. Your clients should not share users, data, pages or anything. You even probably want to have different URLs for different clients for example:

client-a.app.com
client-b.app.com

One way of doing that is duplicating your app instance for each client. Each instance can be put on different machine or they can share one physical machine but can be separated from each other using virtualization or using containers.

But hey - what if your app gets popular and you get thousands of clients? Does it mean you need virtual machine/container or real machine for each of them? Well... yes. This approach of splitting your clients is called "Single-tenancy" (or "Singletenancy" without a dash). Another approach is a "Multitenancy" (sometimes referred to as "multi-tenancy" with dash or "multi tenant"). This approach, in simple words, means to have different clients on same server. The clients share the same database and all the services. The clients are only separated from each other on database level by using for example some column which distinguish them.

I actually got some questions about multitenancy in Liferay from one of the clients I did some consulting for and I thought I will cover part of this topic in an article.

Note some sources also mention something "between" these two where there is a single app like in multitenancy approach but uses different databases (or database shards) for each client like in single-tenancy approach. In this article I will not talk about this approach too deeply. I will just give you some informations about Liferay support of multi databases in the end of this article. Beside that I will stick only to single-tenancy and multitenancy approaches for the rest of the article.

Single and multi tenancy comparison

So now you have the basic understanding of what single and multitenancy is. Lets compare some more aspects of these two.

We will start with things that are better with single-tenancy approach and then we will move to the aspects that are better in multitenancy.

Warning please note that I will not go through every aspect that might be important. There are other articles that might cover the topic more deeply. In this article I want to introduce you to the topic but more importantly I want to show you how multitenancy can be achieved in Liferay.

Data separation and security that comes with it

Probably the main benefit of single-tenancy over multitenancy is the fact that the data is really separated from each other. Even if there is some security issue with your app, client A will not see the client B data.

With multitenancy each client is on the same database so if there is any flaw in your app there is a chance that clients will see eachother data. This of course would be a huge issue for any application provider - not only because app doesn't look to good to its clients but also it might come with some legal issues for example with EU GDPR.

Different clients can have different versions of your app

Another benefit of single-tenancy might be the fact that you can have slightly different versions provided to each client. In some rare occasions it might be crucial for your app to have different modules versions for different clients. This approach is not recommended as it might get really hard to manage these different versions in the future. Nevertheless if you have just couple of clients then you might want to use that sometimes.

In multi tenant approach you have one server so you provide everything or nothing. This of course doesn't mean that you can't prepare per client solutions. You should simply consider other ways of providing these for example using so called "Feature flags". In Liferay's portlets you can also simply deploy your new portlets but don't put them on any page or even disable them per client (we will go into details of this later).

Deployment process

In single-tenancy deployment process will be more difficult. Of course as the number of clients increase you will need a good automatic deployment process (continuous delivery/deployment) but it still might cause you some issues if anything goes wrong and you have hundreds of clients (machines, virtual machines or containers).

With multitenancy there is one single server (or cluster) so deployment process is simplified but it comes with a cost actually - if there is an issue in your deployment it will affect all the clients. This is the aspect where both single and multi tenancy have their pros and cons.

Resources

Single-tenancy doesn't shine so brightly when it comes to resources though. Since you use container/virtual machine or even real machine for each client then you need more resources. Of course Docker is better solution than real machines when it comes to resources increase but it still comes with some cost.

With multitenancy you have only one server (or cluster of servers) so you can manage that better.

Providing high availability (HA)

Providing high availability (HA) in single-tenancy approach might be harder as each client has its own instance of an app - it might be impossible to create a cluster for each one of them as it would be expensive and hard to manage.

With multitenancy clustering is something you will just need to do at some point as more clients means more resources and and some point you will encounter limits of your server.

Multitenancy in Liferay

Now we had a little introduction to the topic of multitenancy lets see how this can be achieved in Liferay. In fact this is quite easy as it's supported out of the box in Liferay Portal with so called Virtual Instances. Virtual instance is just a virtual thing that lets you split your application. Each virtual instance has its own URL, its own users, organizations, roles, pages and even its own sites. There are also lots of per instance configurations.

Note I will use Liferay 7.4.0 CE in my examples but the settings I will show are quite similiar in all of the versions of Liferay, even in obsolote 6.2. Sometimes they are just hidden in other places of the control panel.

Instances routing & default instance

You might wonder - how Liferay knows which instance user should see when he enters your portal? Well it's quite simple. Each instance has it's own domain, or so called "Virtual Host". If user enters your app by this domain Liferay knows which instance should be served.

Another important term is a "Default instance". What is a default instance? Well by default it's an instance created automatically when you first started your Liferay Portal. This is also the instance that is shown to you if you enter your portal using IP address instead of the domain name.

Default instance is also like a "root" instance and most priviliged one - some config can be made only there for example changing system settings or configuring virtual instances. Because of these two you probably don't want to put any tenant on your default instance and simply use it as an administrative one. You can also consider blocking access to your portal by IP so no one has accidental access to this default instance.

If you use Liferay in single-tenancy way (with only one client) then you probably can stick to using only default instance.

It's also important to know that user can't check all the instances you have created unless he has administration role in your default instance.

Changing default instance

Sometimes you might need to change the default instance of your Portal. This option is a little bit hidden and I couldn't easily find any informations regarding that in the Liferay's docs. This is possible though using portal.properties config and the key you're looking for is "company.default.web.id" with default value set to "liferay.com"

company.default.web.id=liferay.com

I assume you're familiar with portal.properties. If not then you can read some more about it, and its extension in Liferay docs.

Instances panel & creating new instance

If you want to see all of your instances you need to go to the:

Menu button -> Control Panel -> Virtual Instances (under System)

First open menu button

Menu button

And then in the panel that shows up select Control Panel -> Virtual Instances

Control Panelu Virtual Instances

Note this page is only available on default instance of your Liferay Portal.

Once you open your virtual instances panel you will see table like on the screen below:

Virtual Instances panel

To create new instance you simply click the blue plus button. New page will open:

Adding new Virtual Instance panel

There you simply need to fill few fields:

Web ID - something unique, as it's the id of the instance. Often it's just a domain name
Virtual Host - the most important field which tells Liferay which domain should be redirected to this instance
Mail Domain - used for emails notifications from the instance
Max Users - maximum number of users for this instance. You can use 0 for "unlimited"
Active - either the instance is active. If it's not then it's not accessible by anyone

After filling everything you confirm using "Save" button.

To test your new instance you need to go to the Virtual Host domain. If you don't have real domain for testing you can use so called "hosts" file to redirect some fake address to your local machine. You can find more info on that topic for example in this article.

In this case that's exactly what I did - I changed my hosts file to point the app.pydyniak.com to my local machine:

127.0.0.1 app.pydyniak.com

The default user is called "test" and its password is also "test" and the email domain is the domain you used for "Mail Domain" field during your instance creation.

You might also notice that your admin user from default instance doesn't work on your new instance - like I mentioned before almost everything is separated per instance and that includes users, even the admins.

From now on you can start using your new instance totally independently.

Things shared between instances

Of course there are some things shared between instances. Probably you can thing of some based on the previous part of the article but I will also point few most important:

Database - all instances share the same database (later I will also write few words about using different databases/shards per instance)
Server - obviously if you use one portal you use same server. Virtual instances are just a partitioning capability
Installed modules/themes - but OSGI portlets, themes and layouts can be disabled per instance. I will go into more details of this later on.
System settings - the top level settings in Liferay are the system settings. They're common for all the instances but some of them can still be overriden per instance and they only act as default unless overriden
Marketplace plugins
Gogo shell - as modules are common so is the gogo shell. It includes both gogo shell available through control panel as well as the gogo shell access using your terminal
Search engine - some search can be configured per instance (fields mappings) but in general same search engine (Elasticsearch) is used across all the virtual instances. Note that fields mappings are configured per instance but the configuration can only be made on default instance.
Log levels

Things above are probably the most important things that all virtual instances share. It's also important to know that in general the options that are shared are only configurable through default instance.

Things not shared between instances

Almost everything not mentioned in the previous point is not shared between instances. It includes:

Sites
Pages
Users, roles, user groups, organizations
All the content including web contents, templates, structures, categories, tags, widget templates, fragments, documents & media and everything else you can think of connected to content
A lot of instance settings (under control panel -> Instance settings)
Active components - I will go into more details of this in a while

Disabling Liferay components

Like I said before: in Liferay OSGI modules are shared between all instances. This is connected to the fact that one server has one common place where it stores all the modules (OSGI ones, themes, layouts etc). So if you deploy something new to your server it's automatically available on all virtual instances. Of course the fact it is available doesn't mean you have to use these. You can simply not put your portlet or any page or don't set your theme for any page.

They're still visible though but there's is a solution for that. If you decide that you don't need some components you can actually disable them and that can be done per instance. To do that you need to go to control panel -> components panel of your instance. In this panel you will see three tabs:

Portlets
Themes
Layout Templates

In each of these tabs you have a list of components with their status:

Components list

You might notice that sometimes there are two components with same name like "Accounts" on the screen above. To get some more details you can go into details by clicking on the three dots on the right:

Component details

And in that panel you can check the Plugin ID to see what is the difference between components. You can also disable the component in that panel by unchecking "Active" checkbox and you also a little summary of permissions connected to this component:

Component edition panel

Likewise you can go to the details of themes or layouts.

There are many different scenarios where you might want to disable some of the components. For example:

You want to disable some default Liferay portlets you don't need - for example you might not need Liferay Commerce modules in your intranet instance
In your SaaS you might want to disable components client didn't pay for so he can't find them and put them on the page without your knowledge - of course make sure that client doesn't have admin account and can't go to components panel himself :) You can always create a "partial admin" role without this specific permission
You have only one or two themes/layouts you want to use across all your instance and you don't want someone to use something not meant for the instance

Please note that of course OSGI modules are still shared and you can only disable some types of components. In some cases you need some extra logic in your code to make sure that it doesn't affect all the clients and only the ones you choose. Such examples can be CRONs or Servlet (Portlet) filters.

Per instance database/sharding support in Liferay

I will not go too deeply about sharding as this is totally out of the scope of this article. If you don't know what sharding is you can use your favourite search engine to find out some more or, if you are not interested in using different databases with one portal, you can simply skip that part of the article.

Some people might wonder if there is any out of the box sharding or multiple databases support in Liferay. If you look for that on the internet you might actually find some old articles how to do that but unfortunately they're outdated. For the moment there is no functionality like that in Liferay 7. Back in the 6.x versions you could define multiple database sources and for each instance you could pick which one to use. For example instances A, B, C could use one database and instances D, E, F could use another one.

This is no longer possible as stated in docs.

Instead you need to stick with one database source and use sharding/replication options provided by your database vendor.

Other use cases for Liferay's Virtual Instances

Please note that achieving multitenancy is just one of the things you can do with Virtual Instances. Depending on your business goal you might have other reasons to use them even if you use it only for a single company. In most cases though sites are the natural choice for single company projects.

Conclusion

I hope this was not too long and at least a little bit interesting. Quite often in Liferay many topics, even the popular ones are not covered enough by the documentation and if you look for them you might find a lot of outdated articles. This is actually the case with multitenancy topic - a lot of articles are written for Liferay 6.x and there are not too many resources about 7.x. Of course you don't want to use Liferay 6.x anymore and there are lots of reasons for that. If for some reason you still use it you should definitely consider updating it - I wrote some articles about the issues you might encounter in Why migrating Liferay projects to a newer version is not easy article and the second part of it. Also if you need any help with migration you can always contact me - migrations are my most popular consulting projects.

I hope this article will help at least some of you.

GraphQL as an alternative to REST API

Mon, 29 Mar 2021 00:00:00 GMT

Hello!

Few days ago I had an opportunity to do a presentation on MeetJS meetup which is basically a series of meetings of JS/frontend community in Poland. The topic of the presentation could be translated to "GraphQL as an alternative to REST API".

The image from meet.js announcing my presentation at the meetup.

This was the first time I did presentation on public conference/meetup so that was an unique experience for me and I'm really glad I had this opportunity.

Because of this event I also thought it will be nice opportunity to write a blog article on this topic. In this article I will explain what is the GraphQL, what are the benefits of using it and what are downsides. I will use Rest API as a reference in some points. If you don't know REST already then you probably should catch up and learn a little bit more about it as it's the most common architectural standard connected to transferring data between systems. Explaining the details of REST Api is out of the scope of this article so if you're not comfortable with it then you should start with some other resources and then come back here. I will assume at least basic knowledge of REST Api, huge experience is probably not needed though.

But what is wrong with REST Api?

But lets start with REST Api and a little explanation what is even wrong with it and why should you even consider any alternatives.

Overfetching

First issue you might have is overfetching. Overfetching is basically any situation when you get more data than needed. For example you need a name of some user but the only endpoint you have available returns also a lot of unnecessary data:

https://www.howtographql.com/basics/1-graphql-is-the-better-rest/

In the example above we also get id of the user, address and birthday. In real life scenario this could be even worse - sometimes when you use headless CMS you might have for example around 30 different fields when you only need one. This of course means more bandwidth used and in some cases more work needed by the server (like additional DB calls). Both results in worse performance.

Underfetching

Second issue you might encounter is underfetching. In previous point we fetched too much data and in this, as you might guessed, we download not enough.

Lets imagine we need to create a small part of the page where we display:

User name
User's posts
User's last three followers

Unfortunately in REST this will often means that we need to perform three requests like on the image below:

Image from https://www.howtographql.com/basics/1-graphql-is-the-better-rest/

This is connected to the fact that in REST each resource (or set of resources of given type) is represented by some specific URL. That's why we do a different HTTP Request for:

User data
User's posts data
User's followers data

This is not the perfect solution for performance reasons. Not only we will have three HTTP requests but also server will have no chance to optimize the process of preparing data.

N+1 issue

The problem that also comes with underfetching is the N+1 problem. You might have heard about it as it's quite common issue related to fetching data from databases but it's almost the same in REST architecture.

Example of this can be following:

You have an /books endpoint which only returns ISBN of the book
In order to get more informations about the book you need to call another URL like /books/{ID}

Look at the image below to get an idea:

https://restfulapi.net/rest-api-n-1-problem/

So if you need to display a list of 100 books with name for each one you need to perform:

One /books request - in order to get ISBN for each book
100 requests for /books/{ID} to get the name for each of the 100 books

Which sums up to 101 request for 100 books and that's why it's called N+1 issue. You need to call another endpoint for each resource. This is an issue for two reasons:

There are 101 HTTP requests each time someone enters your site. Even if backend responds really quickly and you have cached response it's still a problem since each HTTP request takes some time
You don't give your backend server a chance to fetch the data in the best way. It doesn't know about 101 requests you're going to make so it will just fetch a data separately one by one for each request.

Coupling backend and frontend

Someone could argue that these are not that big issues. After all you can simply ask your backend developer to make an endpoint that returns exactly the data you need, can't you? Well you can but the issue is that it couples backend and frontend together much more. The coupling in software engineering tells you how closely modules or part of your system are connected or, in other words, the strength of the relationship between them. Why is high coupling bad? Well because each time you change your module other modules that are highly coupled need to be changed as well.

In our example - if we tell our backend developer to make an API that returns exactly the data we need that's fine until there is a need for a change. Lets imagine that we have the application which displays 100 books (as in example above) and we asked our backend developer to make an API with the exact data we need. So we have something like:

GET /books - returns books we need and for each it also returns the name

This way we don't need to make another call for book details and we avoid n+1 issue. That's good but then client comes and asks for another field - now he wants to include the book publishion date in the books list. In such scenario frontend developer again needs to ask the backend developer and they both need to do changes and synchronize them and also make sure they're deployed at the same time. This is the issue we have from high coupling between parts of our system.

Another issue might be that we can't always change the API easily. There are many cases where it's hard. For example:

We have some API developed by third party company and it's expensive to make such changes
We use some public API for example for some government data or stock market data.

In such scenarios we're stuck with N+1 issue and performance issues.

GraphQL

Now we understand what issues we might encounter while using REST lets finally get to the GraphQL. I know it was quite a long introduction but I believe there's no point of showing an alternative without explaining the issue first.

Few words of a history

Well I don't know if you remember these times but apparently back in 2012 the Facebook mobile app wasn't as good as it is now. It used to crash a lot and the performance was poor. The reason for that is that mobile apps were simply a wrappers for mobile website and therefore the possibilites for improving them were limited. In that year the Facebook team decided to rebuild their app and improve it. They tried different options for API including REST but eventually they started work on a thing that now is known as GraphQL.

For details of the history you can check the post on Facebook blog.

So what is GraphQL?

GraphQL is simply a query and data manipulation language for APIs. Like REST it's not tied to any database, storage, language or framework. Instead it's more like an approach for your API development. You could say that GraphQL is a presentation layer for your backend, exactly like REST. GraphQL is traditionally served over HTTP but it's actually agnostic to the transport layer - you could also use different method of connection like WebSockets. The HTTP is the most important though and I will only focus on it.

There are features of GraphQL which makes it different than REST though.

Declarative data fetching

Declarative data fetching is probably the most important feature of GraphQL. Remember when we talked about issues with REST? They all came from the fact that our API had defined endpoints with defined data these endpoints return. Because of these we had our overfetching, underfetching and N+1 issues. Because of these we had a choice either to go with poor performance or high coupling between frontend and backend.

Declarative data fetching is a response to that issue. When I talked about underfetching I presented an example with simple part of page with name, followers and blog posts where we needed to make three REST requests to get the data we need. We also got more data we needed for each of these endpoints. With GraphQL our API call could look much simpler:

https://www.howtographql.com/basics/1-graphql-is-the-better-rest/

If you take a closer look on the request and response you might notice that we basically defined data we need and our API returned exactly what we wanted. This is declarative data fetching - it gives us a possibility to define what data (fields) we need. We also don't need to call different URLs for different kind of resources, instead we can call for user and in the same request we ask for different resources connected to the user like posts and followers.

It's a powerful feature and actually it solves all of our issues with underfetching, overfetching, N+1 issue and high coupling.

Coupling

Remember how I said that you might encounter an issue for example with government data? Or with data from another company? In such scenarios it can be really costful or even impossible to change the data returned. If such public APIs used the GraphQL these issues would not exist as all clients could simply ask for data they need.

There is only a single endpoint

One of the thing about GraphQL is that there is only a single endpoint available which is like a proxy for all the data. Instead of calling different endpoints for different data you always call the same one and just describe what you want to query or modify. You can say that GraphQL acts as a proxy not only because you call single URL but also because you can ask for data from its database or from other microservices, systems or APIs

https://www.howtographql.com/basics/3-big-picture/

We can define how the data should be retrieved by the server using "Resolver functions". These functions job will be to retrieve the data user asked for. In many cases it will simply call your database (or more likely, use the service layer) but just keep in mind this is not always the case.

Strongly typed

GraphQL is strongly typed. The types for each field are defined used Schema Definition Language (SDL). Of course if you come from languages that are not strongly typed you might think that it is something you won't need but remember the APIs are meant to be used by another people. It's not just some simple function only you will write, change and use.

The API might be used by different developers, perhaps not from the same company. The fact that the API is strongly typed is helpful for making a contract between clients of the API and the developers who create that API.

Response mirros the request

Like you could see above - the response from the API looks very similiar to the request. This is really useful if you think about it as the returned data is really easy to predict.

Introspection

In GraphQL you can use a feature called "Introspection" for describing the API. This means you can simply ask the GraphQL API for defined resources and types:

https://graphql.org/learn/introspection/

This feature can be used by the GraphQL client you use so it will automatically generate a documentation for you - without any additional work from the developer! Check the GIF below for an example of such generated docs:

Mutations for data manipulation

Usually API not only returns data but can also be used for inserting, modifying or deleting the data. In REST you do that by using different HTTP methods: POST, PUT, DELETE and sometimes PATCH.

In GraphQL on the other hand you use a thing called mutations. You can think of mutations as defined methods you can call. In REST you have different URLs combined with HTTP method for different thing you want to do. Here you have different methods under different names you can call. Below you can see an example of calling such method

https://www.howtographql.com/basics/2-core-concepts/

You can also notice that not only you call method with some parameters (name: "Bob", age: 36) but you can also define what data you want to be returned when the record is created or modified - just like in queries.

Of course available mutations are also available through introspection and therefore in automatically generated docs.

Downsides

GraphQL like anything else had its pros and cons. If we talk about downsides there are few most important.

Can't we do the same with REST?

Someone after reading all of these could ask - well that's cool, but can't we do the same with REST? For example use HTTP parameters to ask for the fields we need:

GET /books?fields=name,publishedDate

And indeed we could. This is not a standard though and we might not find that in APIs created by another people. Also it could work well with simple fields but it would be much harder to ask for complex data types - for example ask for User and for his followers (which are also User type) and to fetch only name from the followers.

But indeed in some cases we might simply not need that at all - for example if we have simple microservice used only by other microservices in our internal architecture.

Adds complexity

For simple APIs the GraphQL might be just too much. It can add complexity which is unnecessary in many cases.

Caching might be harder with GraphQL

In REST we have endpoints. Each endpoint represents some data. In GraphQL on the other hand we have one endpoint and the returned data is calculated based on what request we make to that endpoint.

If we have some cache layer between our server and the client then it might be an issue as caching whole URLs is just easier.

It doesn't mean we can't have a good performance using GraphQL - we simply need to think a little bit differently about the way we use cache and the connections between layers in our architecture.

Performance for multi level queries

Since GraphQL gives you possibility to connect data the way you want you might be tempted to ask for data too "deeply". For example ask for author of blog post, then ask for all of his posts then ask for all reviews and for user of these reviews etc. Such use cases might result in poor performance on the backend side as querying for the data could become an issue.

Less popular

REST Api is way more popular than GraphQL. This means more resources, more frameworks, most questions on StackOverflow. Your job colleagues also might feel more secure to use something they already know than learning something new.

Of course GraphQL isn't that new anymore but still there is a huge gap between REST and GraphQL when it comes to popularity. On the other hand you will already find quite a lot of resources on the topic and frameworks which supports it. In Java for example you can easily create it using Spring - it's not that different from creating REST Api.

Anyway - will it replace REST?

Well there is no single answer. GraphQL is not meant to be replacement of REST but an alternative. Anyway in my personal opinion in some cases it's simply better. Examples of such scenarios can be:

API for different clients for example Android+IOS+Web app - in such scenario each client can make changes in their own pace or use the API slightly differently - it's quite common to have slightly different data on Web App and mobile app
Publicly available APIs - you don't know your clients. You don't know what use cases they might have with your data. In such scenario you can't provide some specific API and instead it must be generic so everyone can use it. If you use GraphQL your clients can fetch the data they need and also avoid issues with underfetching/overfetching
Using API available in CMS - in some cases you might have an option to pick between REST API or GraphQL provided by CMS you use. This is for example the case if you use Liferay - both REST and GraphQL are available if you plan to use it as headless CMS for your data. In such scenario I believe you should go for GraphQL as even if REST seems fine at first it might not be fine later when the use cases are more complex

There are also scenarios where REST might work just fine. For example:

Simple server with not too many endpoints
Microservice for internal architecture

Summary

To summarize - I believe if you ever worked with REST you should also know the basics of GraphQL. This way you will be able to make better decisions about architecture of your application. This will of course result in better app you do, better API you provide to your clients or perhaps reduces time to market and simplifies the maintainance of your app once it is realeased.

In this article I didn't touch any code. I did it on purpose as the code will look different for different languages/frameworks but the concept will stay the same. I haven't also explained too much of the GraphQL code but that's simply because there is a nice documentation available at https://graphql.org/learn/. I belive you will find everything you need there once you decide to learn something more about GraphQL or you decide to start using it.

Like always I hope you learned something. If you had any questions you can always contact me or leave a comment.

Why should you use CMS such as Liferay for your web application

Fri, 12 Mar 2021 00:00:00 GMT

In this short article I would like to answer the question that some of you might have "Why should I use CMS for my website". As an example of such CMS I will use Liferay which is primary system I use.

What is CMS anyway?

Lets first define what CMS system is. CMS is just an abbreviation for "Content Management System" which is a software that simplifies the process of creating and modifying digital content. The main two areas for which CMS systems are used are:

Web content management - creating web content such as pages on your website or articles on a blog
Enterprise content management - digital documents management in a collaborative software

Beside these two there are many other things that typical CMS supports such as:

Managing the digital assets like videos or images
Users, authentication, roles
Multiple languages
Administrator panel for non technical users
Content hierarchy which means that pages can have subpages, images can be grouped into directories etc.
Tools for improving SEO of your site

Of course there might be much more features (or less) depending on the CMS software you use.

But are the CMS systems actually popular?

At this point you might wonder are the CMS systems popular at all? Well actually yes. According to W3Techs.com five most popular CMS systems have almost 50% of the market share:

Image showing market share by top 5 CMS

As you can see the number one is definetelly Wordpress which has around 40% market share.

Well should I use Wordpress then?

You might wonder - if wordpress is so popular, should I use it as well? Well it depends. If you need a simple personal blog or a website for small company then... yes! You should definitely consider Wordpress as your CMS system. Although if you need more complicated website with your own modules, advanced permissions system, best performance, clustering etc then wordpress might just be not enough. Wordpress, like everything else, has its pros and cons and the main property of wordpress is its simplicity. While it's advantage for small sites it also might be a huge disadvantage for bigger ones. For example if you want your site to have a lot of users with different roles and permissions or you need to synchronize data with third party system like SAP or perhaps you want to develop your own application for your employees and you need a lot of custom modules then it might just be too simple for your needs.

What are my options then for more sophisticated website?

Actually there are quite a lot of advanced CMS systems and it would be hard to present any details about all of them (or at least half of them) and that's why in this post I will only focus on one of your options and perhaps in the future I will also write a blog entry about some others.

Today I will focus on Liferay though.

What is Liferay then?

Well since the previous part of this article was focused on CMS systems then... well you guessed correctly, Liferay is one of them. Unlike Wordpress it's more complicated though and it also gives you much more possibilites when it comes to creating and managing your website.

Does it mean you can't use it for a simple website? Of course you can! You just probably don't want to :) Let's then talk about few properties of Liferay or, in other words, reasons why would you want to consider it or not as your web platform. Of course it would be hard to go through all of them and that's I will just mention few - of course in the end every platform needs to be considered in details for specific use case (in which I can of course help, just write to me and I will try to do my best!).

Advantages of using Liferay

Lets first go through advantages and then I will also talk about disadvantages

Advanced pages hierarchy and documents & media/web contents segregation

Liferay is quite great when it comes to configuring your webcontents, pages or your media. You can have folders with different media types or different web contents. You can have pages and subpages. Actually everything can be organized which is great for huge sites with thousands of different kind of assets. Of course you can also categorize your content and use tags, whatever you need.

Liferay can also handle searching through different kind of assets you might have in one common way. You can easily choose for which kind of assets your search box should look for. You might for example only look for pages or you might want to search your whole system (pages, users, images, blogs, etc) - it's totally up to you but Liferay gives you a lot of options that you can configure.

Advanced users & permissions system

Liferay has advanced permissioning system. Almost everything in Liferay has its own permission so you can easily decide what exactly you want users to see or be able to do. If you develop your code it's also easy to create your own permissions so then they can be easily managed from the control panel without any technical knowledge.

Another important thing is the number of possibilites you have to organize your users. You can create multi level structure of users using organizations or you can simply group your users using a thing called "User groups". Or you can use them both to fill your needs in a best possible way. After you organize your users you can then easily assign them to different roles or sites to give them access exactly to things you want.

But what if you already have your users in some system? Well the great thing is that you have out of the box integration with different, most popular systems (LDAP, SSO, CAS, Facebook and many more) so perhaps you don't need even a line of code to integrate users between systems.

Multiple instances or sites on same server

Well imagine that you're creating system you plan to sell to different companies. For example a system in which company's employees will be able to create holiday requests, organize their worktime and stuff like that. For obvious reasons you don't want to mix these users on same server, they should not see each other. Company X should not even know that company Y is also using the same system. Of course they also want the urls connected to their company so for example x.com and y.com. On the other hand if you have a lot of clients then creating another server and then managing it for each of them might be a hard and expensive process.

In such scenario Liferay virtual instances might be handy for you. Each client might be on his separate instance. Each instance has its own pages, users, roles, content and even its own url. Basically everything is separated which decreases the costs and also might cut time to market since you don't need different server, you just need to create new instance for your new client.

But hey! That's not all. The company X might want to have different sites for different kind of employees and you can also easily do that. So each client can has its own virtual instance and on each of these instances you can have as many different sites with different pages as you need.

Basically whatever you imagine is probably possible.

Clustering support

As the number of clients increase one server might be not enough. That's fine though as Liferay has "native" support for clustering. I wrote about clustering in Liferay before in Step by step guide for creating clustered Liferay environment article and therefore I won't go into too many details but lets just say it's simple. Because of the out of the box support for clustering it is not only easy but also gives you different configuration possibilites - you can do it almost automatically with multicast but if that's not an option (for example because servers are not in the same network or because of the cloud limitations) then you have different options like TCP unicast.

No matter if you need clusters in Docker, Kubernetes or just few machines in the cloud - you can handle that quite easily.

Custom, reusable parts of the page with portlets and custom modules

Of course Liferay won't handle all of the business needs you might have now or in the future. In such case you can simply create your own modules in Java. Liferay has support for many things you might need like database access using ServiceBuilder, creating the service layer, creating well structured code base with OSGI modules which also makes it possible to deploy modules without the need of restarting anything. You also have different possibilites of creating the presentation layer.

Liferay uses portlets which are modules that can be put on any page of your site and they act as a frontend which your users will see. Portlets are standarized thing in Java, described by JSR-168, JSR-286, JSR-362 specifications. Because of this it's also possible to use for example Spring MVC portlets.

Basically creating modules is a huge topic but lets just say you can do quite a lot.

Extending Liferay

In some cases it also might be crucial to extend your Liferay to your needs. For example you might need to change some translations, modify view just a little bit or add some extra logging when user login or logout. There are tons of different scenarios such as these and actually Liferay supports extending quite well. You don't need to modify framework code but instead you have different ways of extending or modifying your Liferay in a way that won't break once Liferay is updated. Because of that it's not only easy but can be done in a safe way.

There are also some extension points which are available directly in the control panel - for example you can add a custom field to things like Users or Sites and then you can use these custom fields in your custom code to meet your needs.

Disadvantages

Complexity

Liferay is great because of number of options it gives to you but because of this it's also more complicated. Like I mentioned in the beginning - you probably don't want to use Liferay if you just need a site with few static pages.

Of course if now you only have few pages but you plan to grow then you might want to start right away with CMS like Liferay, just keep that in mind and if you can consult your decision with someone more experienced.

Server resources

For a simple site (like this blog) you don't need too much server resources. Even something like 300mb of memory might be sufficient for you needs. You can even just buy a server with already installed Wordpress or has a script for installing it with few clicks. In Liferay you will need much more resources even for an empty portal which means more expensive server. Also you will need to configure it on the server yourself, along with the database or you need to pay someone to do it for you.

Again, these server costs are not that big but it's just another reason why for simple site something else might be enough and cheaper.

Wordpress like plugins/themes

If you have used Wordpress before (or any other popular CMS) you might know that you can do tons of things with already existing plugins. A lot of these plugins are also free which is also great. That's sadly not a case with Liferay. There are tons of already available options so for many cases you will be fine but if you wanted something less popular (lets say to add some SEO stuff) then in Wordpress you had a chance to find a plugin which does that for you. The marketplace in Liferay on the other hand is quite small so if something isn't available in Liferay then you will probably not find an existing plugin for it. In such scenario you need to find someone who will write required module for you.

The same thing is true for themes. For Wordpress you will find thousands of them, a lot of which are free. In Liferay you will only find few.

But like I mentioned before, Liferay is not meant for simple pages so custom development is most likely thing that you just need to have on your mind.

Summary

To sum it up. In many cases you should consider using CMS for your website. Depending on your needs it might be the best to use simple one like Wordpress or Joomla or you might consider something more advanced like Liferay, Kentico or Adobe Experience Manager.

Each of the solutions has its own advantages and disadvantages. I hope I gave you at least a little understanding why you should also consider Liferay in your project. Of course I mentioned just a few properties but there are much more and there are many uses cases for it. Sometimes you will use as a bank site, sometimes you will just keep content in it and use it as a Headless CMS and sometimes you might want to develop a whole SaaS with it. Because of so many uses cases and options in Liferay it would be hard to list them all and therefore it's always good to ask someone more experienced to help you find a tool that will work best in your scenario.

Step by step guide for creating clustered Liferay environment

Tue, 26 Jan 2021 00:00:00 GMT

Hello! In this article I would like to show you how to create clustered Liferay environment. In my example environment I will use docker but the steps are similiar if you're installing two (or more) instances on one or many servers.

I will try to show you step by step how you can build the environment yourself so you can understand the process better. Of course if you want to you can go to the final version of code right away - it's on my git. I encourage you to go through the steps and learn something new though!

Important thing - I will focus on simplicity and on Liferay configuration so I will not create database/Elasticsearch/loadbalancer cluster. If you need those then it's quite simple to add that to the basic configuration I will show you and of course you will find tons of resources on these topics. Instead I will just show you the basic configuration for database/ElasticSearch and load balancer.

Prerequisites

To start you will need Docker installed on your local machine (or server or any other place where you want to start your cluster environment).

I won't go through the basics of containers or docker so it would be good if you knew the basics but I guess anyone can follow.

If you plan to create your environment while reading then you can also create a new directory with files/directories like on the screen below (the files can be empty at this stage):

Initial files structure

And in the deploy directory you can create two empty directories called "portal-node-1" and "portal-node-2":

Deploy directory initial files structure

Or you can check the GIT repository instead if you prefer to see the end result from the beginning.

Docker storage - mounting, volumes

After publishing the original post I thought about one more thing - in below instructions I use Docker volumes for holding some data (elasticsearch and Liferay's documents repository) and I mount some other (deploy directories) and I also copy some (configs, jars, portal-ext.properties) in Dockerfile. Also for the MySQL I don't save the data anywhere actually.

This approach gives you an overview of different methods you can use but there's a catch. If you plan to create production environment based on the instructions below then you should understand data storage in Docker first! I believe the best place for that is Docker documentation. It's really crucial as all types of data storage methods in Docker have their pros and cons.

Process of building the environment

Liferay containers

Since we want to have clustered Liferay environment we need two or more Liferay instances. I will go for two but it shouldn't be a problem for you to extend that number.

So how do we start? Well since we want to use docker compose we could create a simple docker-compose.yml file with description of two Liferay containers but I won't do that. Instead I will start with "Dockerfile-liferay" with following content:

FROM liferay/portal:7.0.6-ga7
MAINTAINER Rafal Pydyniak <rafal@pydyniak.pl>

COPY ./configs/portal-ext.properties $LIFERAY_HOME/
COPY ./configs/\*.config $LIFERAY_HOME/osgi/configs/
COPY ./configs/\*.jar $LIFERAY_HOME/osgi/portal/

So basically we create our own image from existing one (official Liferay 7.0.6 GA7 image). In our version of image we copy portal-ext.properties to LIFERAY_HOME directory and also config files and some jars. We will need that later. Of course we could map our local disc to docker disc instead but since these files are not going to change we will do it this way. Perhaps the portal-ext.properties is an exception as it changes more often but I will leave it this way for now.

If you want you can test it now. To do that build that image, start it and then connect to the container to see if the files were indeed copied and the directory was created. Following commands will help us with that:

docker build -t my-liferay-7-tutorial -f Dockerfile-liferay . # builds the "Dockerfile-liferay". The created image will be "my-liferay-7-tutorial". The dot at the end is for pointing the context directory
docker run -d my-liferay-7-tutorial # we run our image
docker exec -it CONTAINER_NAME /bin/bash # we enter the container so we can test if everything is there. You can find the CONTAINER_NAME using docker container ls command

Use our images in docker-compose.yml

The next step is to create our Liferay containers (instances) definition in docker-compose.yml. The first scratch of the docker-compose.yml is below:

version: '3.3'
services:
  liferay-portal-node-1:
    build:
      context: .
      dockerfile: Dockerfile-liferay
    ports:
      - '6080:8080'
      - '21311:11311'
    hostname: liferay-portal-node-1.local
    volumes:
      - lfr-dl-volume:/opt/liferay/data/document_library
      - ./deploy/portal-node-1:/opt/liferay/deploy
    depends_on:
      - mysql
      - es-node-1
  liferay-portal-node-2:
    build:
      context: .
      dockerfile: Dockerfile-liferay
    ports:
      - '7080:8080'
      - '31311:11311'
    hostname: liferay-portal-node-2.local
    volumes:
      - lfr-dl-volume:/opt/liferay/data/document_library
      - ./deploy/portal-node-2:/opt/liferay/deploy
    depends_on:
      - mysql
      - liferay-portal-node-1
      - es-node-1
volumes:
  lfr-dl-volume:

So what we did there? Well we defined two instances of Liferay "liferay-portal-node-1" and "liferay-portal-node-2". We use our dockerfile we created before as the image and use current directory as a context path. We also expose ports from our container so they can be accessed outside the container. So we expose port 8080 to 6080 (node-1) and 7080 (node-2) and we also expose the gogo shell port.

Then we set the hostname so we can distinguish our instances and then we use volumes to do two things:

persist the documents & media data to docker volume (defined at the end of the docker-compose). This way we have shared data directory between two instances as we use the same volume and that is what we need according to the documentation - all instances should share the same document library.
map our local deploy/portal-node-X directory to the deploy directory of the container. This way we can easily deploy our modules

At the end we also define our volumes - for now it will only be "lfr-dl-volume".

You can also notice the "depends_on" fragment. This is just a declaration on what containers our container depends on. We define that so for example Liferay instance is not started before Elasticsearch or our Database which would result in errors. Of course we don't have Elasticsearch or mysql container defined yet but we will get to it soon.

Documents & data

It's important to add that for production purposes you might want to use "Advanced File System Store" data repository instead of the simple (default) one. In this case I used the default one and the default location to keep it simpler as changing that is not requirement for clustering (which is the main topic of this article). For more details about data repositories you can check this article.

Database configuration

For any Liferay environment we need a database (we don't want to use embedded H2). I won't go into too many details - the block for docker-compose.yml looks like that:

mysql:
    image: mysql:5.6
    environment:
      MYSQL_ROOT_PASSWORD: my-secret-pw
      MYSQL_DATABASE: lportal
    ports:
      - "33066:3306"

As you can see I'm using mysql but you can use any other database. You will easily find images on docker hub for example for postgres.

We also need a defined connection to our database. To do that we will use portal-ext.properties. You might remember that while creating our Liferay dockerfile we included a statement that copies our portal-ext.properties from configs directory to the server. This is the place we can add our database connection declaration. It will look like below:

jdbc.default.driverClassName=com.mysql.jdbc.Driver
jdbc.default.url=jdbc:mysql://mysql:3306/lportal
jdbc.default.username=root
jdbc.default.password=my-secret-pw

Of course you can also add your own properties like you would do in any other case. We will also add some more properties later when we get to cluster configuration.

Note: I actually don't store the mysql database data anywhere so it won't be persistent - if you remove the container (or use docker-compose down) the database data will be lost. To avoid that you can simply use volume (see elasticsearch config for reference) or create another directory and map it there like:

volumes:  
- ./data/mysql:/var/lib/mysql

Elasticsearch configuration

One of the steps for clustering described in Liferay documentation is installing an instance of Elasticsearch. Since we're clustering we can't use the embedded one as all of our Liferay instances need to communicate with same Elasticsearch. In our case we want to use 2.4.6 version of Elasticsearch but in your case you might need another version - it all depends on the Liferay version you choose to use. You can refer to this article for informations how to find the correct Elasticsearch version for your Liferay.

After you find the Elasticsearch version that will work for you, you need to add it to your docker-compose.yml. In my case it looks like below:

es-node-1:
    image: elasticsearch:2.4.6-alpine
    environment:
      - cluster.name=elasticsearch
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - esdata1:/usr/share/elasticsearch/data
    ports:
      - 9200:9200

So again - first we define image we want to use. Then we have some additional configuration options which are either self explanatory (like memory args) or you can find their meaning in google (especially for ulimits). The only important setting connected directly to Liferay clustering is the "cluster.name" - this has to be something unique for your cluster environment but it can be any string. In our case it will be unique as our environment is quite simple but sometimes you might have for example DEV and TEST environment on same server and then you need unique cluster.name property to distinguish them.

You can also notice that we use "esdata1" docker volume so we need add its definition to the end of docker-compose. Our volumes definition will look like below:

volumes:
  lfr-dl-volume:
  esdata1:

Using remote elasticsearch in Liferay instances

Now that we configured our Elasticsearch server we need to actually use it in Liferay. To do that we will usage OSGI's configuration file. Lets create a file called "com.liferay.portal.search.elasticsearch.configuration.ElasticsearchConfiguration.config" in the "configs" directory. Then put the following content inside:

operationMode="REMOTE"
transportAddresses="es-node-1:9300"
clusterName="elasticsearch"

The code above just sets the Elasticsearch mode to REMOTE and sets the address and clusterName. You might wonder why we use "es-node-1" as host instead of something like "127.0.0.1" or "localhost". We do that because the services we define in docker-compose are in the same docker network and they're reachable by other containers by their container name. The binding is handled by the docker itself - it basically adds required entries in /etc/hosts and the container name is just an alias there.

Also note that the "clusterName" must correspond to the name you defined in Elasticsearch service definition.

We don't need to do anything else because in the Dockerfile we declared before we already copy all .config files to LIFERAY_HOME/osgi/configs directory.

Using "multiple" version of bundles

In Liferay 7.0 CE which we use in this article we need to do another step described there. Basically this version by default is not ready to be run in clustered mode because it only contains "non-clustered" version of three modules:

com.liferay.portal.cache.single
com.liferay.portal.cluster.single
com.liferay.portal.scheduler.single

We need to disable them and instead use the "multiple" versions of these. The article linked above describes the steps for obtaining (compiling) them so I won't go into too many details here. I will just describe what to do with them once you get them.

Note: if you use the 7.0.6 version of Liferay you can just get these from the git repository for this article.

Note2: you might not need that step at all if your version already supports multiple module versions. This is true if you use DXP version or if you use newer version of Liferay - just run the gogo shell command in your Liferay "lb portal.scheduler" and see if there is enabled module called "Liferay Portal Scheduler Multiple" - if yes then you don't need to do that step

Once you have your multiple modules you need to:

deploy them - just put them in "configs" directory
ignore the .single version modules - to do that create new file called "com.liferay.portal.bundle.blacklist.internal.BundleBlacklistConfiguration.config" in configs directory with following content:

blacklistBundleSymbolicNames="com.liferay.portal.cache.single,com.liferay.portal.cluster.single,com.liferay.portal.scheduler.single"

The Dockerfile-Liferay we defined before will do the rest i.e. copy them to approriate directories.

Enabling the cluster mode

We are almost done but we still didn't tell Liferay that we want to use clustered environment. How do we do that? Well it's quite simple actually - all we need is to add following line into our portal-ext.properties:

cluster.link.enabled=true

But hey - is that that simple? Well actually it is - at least in some cases. Basically this property makes Liferay to use JGroups channel to communicate with each other on default ips and ports using UDP multicast connection. Therefore it will only work if the Liferay instances are able to communicate with each other through multicast - this is not always the case. Even if you have servers on the same subnet this still might not work as for example Azure doesn't support multicast in virtual networks and they're not planning to add (last time I checked). In our case it will work though as we use docker-compose on one machine and docker will provide multicast connection. If you need other solutions then you can check Liferay documentation where other method is described (unicast over TCP).

For our environment this is enough though. We don't need to do anything else but you might also want to add one more property:

web.server.display.node=true

This will add displaying a message with current Liferay's node. This is useful for testing purposes so I encourage you to do that. Of course you might want to add other properties you need.

You might want to check the Liferay docs to read how exactly the JGroup communication works. The idea is quite simple and clever and it should take you just a couple of minutes to read that and understand the basics.

Load balancing

Initial note

Please note that this step is totally optional. You might already have a load balancer or HTTP server (like Apache or Nginx) installed on your server and then you can't declare another one here and instead you should do the load balancing in your current application or ask your server admistrator to do that for you. If you want (or need) to do it by yourself you can always google it like "Load balancing Apache2" or "Load balancing HAProxy" etc. You can also follow steps below for HAProxy configuration on docker but the steps are similiar if you have it installed as a regular service.

Important you need to have sticky session in your load balancer. This is important as session is by default not shared between cluster nodes so if the user gets redirected to another node he will be logged out. This might be an issue if you use Nginx (which is really popular) as by default sticky sesions are available only in paid version of the app as far as I'm concerned.

Docker config

So the last thing we need is load balancing between our servers. We obviously don't want user to pick the server he wants to enter himself. Instead we want to have one url like app.pydyniak.com (nothing there so you don't need to check :)) which picks one of the servers automatically. The digram below shows the basic idea:

Digram of the architecture

In the example I will use HAProxy but you can use whatever you want as this step is totally independent from Liferay (see the "Note" above).

So first we want to build another Dockerfile. Lets call it "Dockerfile-haproxy" and copy the content inside:

FROM haproxy:2.3.4
COPY ./configs/haproxy.cfg /usr/local/etc/haproxy/haproxy.cfg

There is nothing special in this code. We just copy our config file to HAProxy and this is actually the recommended way shown on Docker Hub. Now lets define our service in docker-compose.yml:

haproxy:
    build:
      context: .
      dockerfile: Dockerfile-haproxy
    ports:
      - '80:80'
    hostname: lb-haproxy.local

We point to our dockerfile and use the current directory as context. We also expose required port and set hostname. I believe it's quite clear.

We also need to fill the haproxy.cfg file we defined in our Dockerfile so we create "haproxy.cfg" inside configs directory and use following configuration:

global
   stats timeout 30s
   daemon

defaults
   log global
   mode http
   option httplog
   option dontlognull
   timeout connect 5000
   timeout client 50000
   timeout server 50000

frontend http_front
   bind 0.0.0.0:80
   default_backend http_back

backend http_back
   balance roundrobin
   cookie JSESSIONID prefix
   server liferay-portal-node-1 liferay-portal-node-1:8080 check cookie s1
   server liferay-portal-node-2 liferay-portal-node-2:8080 check cookie s2

This is quite a long configuration but what is important is that it does what we wanted. The highlighted lines are the core ones. Basically in these line we do two things:

Set the mode to roundrobin - it's just one of "ways" of distributing traffic between servers. You will easily find that on Wikipedia for example
Define the servers available int he cluster and also implementing "sticky session". Sticky session means that if an user enters the site and for example enters the NODE2 he will stay on the same node for the whole session. This is required as the session is not shared between nodes. To understand better how exactly the sticky session is handled you can check this article on HAProxy site

Lets run our Liferay clustered environment

Finally we're done. We defined everything we need and I hope you understand why we did what we did. Now we need to start the environment. We can do that by executing following commands:

docker-compose build
docker-compose up -d liferay-portal-node-1
# wait until it starts. You can also check docker-compose logs to validate that
docker-compose up -d liferay-portal-node-2
docker-compose up -d haproxy

During start of the servers we can see in logs (docker-compose logs) that indeed the nodes connect to each other. You can look for message like below:

liferay-portal-node-1_1  | 2021-01-24 23:10:59.862 INFO  [Incoming-2,liferay-channel-control,liferay-portal-node-1-3920][JGroupsReceiver:90] Accepted view [liferay-portal-node-1-3920|1] (2) [liferay-portal-node-1-3920, liferay-portal-node-2-36188]
liferay-portal-node-1_1  | 2021-01-24 23:11:00.057 INFO  [Incoming-2,liferay-channel-transport-0,liferay-portal-node-1-47303][JGroupsReceiver:90] Accepted view [liferay-portal-node-1-47303|1] (2) [liferay-portal-node-1-47303, liferay-portal-node-2-30594]

If you then enter the server on :80 port you will see that you've been connected to one or two nodes (of course if you added property for displaying that message):

Message which tells us on which node we are

You can also try to connect to the same server from different browser - you will most likely get connected to the second node - if that's the case then congratulations. You can't tell right away that everything works because we only know that:

In theory the connection works between nodes looking at the logs
The HAProxy seems to be working - you can also verify the cookie to see that indeed we have server information there

But in real case scenario you would need to perform some more tests:

If the elasticsearch works - if you add some content on node1, can you search for it on node2?
Does the connection indeed works? If you add some content on node2, can you see that content on node1?
Are the documents & media shared? If you add some document on one of the nodes, can you see it on another one?
If you stop one of the nodes, is everything fine?

And of course you might come with your own tests you would like to perform especially if that's the production server.

Performance

One thing you might notice is that the servers are not too fast. That's because we didn't set memory settings. The simplest way to do that is probably to create configs/setenv.sh. Inside that file you would need to tune the memory settings (override default content). Then you would need to add a COPY in your Dockerfile-liferay. You need to copy that to /opt/liferay/tomcat-8.0.32/bin/ ($LIFERAY_HOME/tomcat-8.0.32/bin/). You can check the git repository where I added that. Please note that you might also need to change the memory settings for your docker so it has enough memory (and CPU).

Conclusion

I hope the article was clear enough. The article was quite long but mostly because I tried to explain a little bit why I did things the way I did. The docker project for that environment is quite small though and you can of course check that on git. In this article I did cluster on 7.0.6 but steps are pretty much the same for newer versions so if you need for example 7.3 then you will probably just do some changes in Dockerfile-Liferay and that should be enough.

If you have any questions feel free to ask either in comments or by contacting me.

Why migrating Liferay projects to a newer version is not easy v2

Sun, 10 Jan 2021 00:00:00 GMT

Hello! In first part Why migrating Liferay projects to a newer version is not easy I introduced you to the topic of Liferay migrations and I showed you a few issues you might encounter during database migration process. In this part I'd like to show you few issues you might have during migration of the code of your custom modules for example portlets.

How to migrate the code

For starters - how do you even start the code migration process? Well first of all if you decided to migrate your project to higher version you should start with the official documentation. There are few options how do you even start migration - for example you will start differently if you're migrating from 6.2 (or earlier) and differently if you're already on version 7.x. Of course the general rule is if you have higher version already then the migration will go easier for you. Also if you use more Liferay's API then it will be harder. Anyway you should start with documentation because it should give you some idea of how it all goes. Obviously after going through that documentation you might feell a little overwhelmed because there're a lot of informations but don't be too stressed about it, it won't be that bad!

Issues that we can encounter

Lets go back to the main thing I wanted to share with you in this post. What can go wrong while migrating the code? Well the biggest issue is that not everything is documented well enough. You can find so called "Breaking changes" pages for each version but sadly not all informations are there and because of that you might find yourself in a place that something stopped working and you can't find a clue why. What can we do? Well I think the only thing you can do is to just go through the migration process and fix unexpected and undocumentted issues one by one. Either by searching them on Liferay forum, or in Liferay blog or just by debugging the Liferay code (I hope it won't be neccessary for you but you never know)

The good news is that I want to show you four issues you might encounter so you can learn something on my mistakes instead on your owns :) These are not all issues I've encountered but I hope that will help you anyway. I also would like to remind you that I wrote about some other issues in previous post.

Note! I found these issues some time ago. It might happen that when you read that (or even at the publication date) some of them are already in breaking changes already - I haven't checked that. This could've happened though as even I wrote some posts in Liferay forum so perhaps someone added that at some point.

Web Contents fetching images API change

I'd like to start by showing you how the XML for image looked like in Web Content in version 7.0:

<dynamic-content language-id="en_US" alt="" name="XXX.png" title="XXX.png" type="document" [fileEntryId="12345"><![CDATA[/documents/111/222/XXX.png/QQQQQQ-WWWW-EEEE-CCCC-DDDDDDD?t=1588696601862]]></dynamic-content>

And then the XML for same (or almost the same) Web Content in version 7.2

<dynamic-content [language-id="en_US"><![CDATA[{"groupId":"126435","alt":"","name":"XXX.png","title":"XXX.png","type":"journal","uuid":"QQQQ-WWW-EEE-CCC-ec48068bbf39","fileEntryId":"716022","resourcePrimKey":"716460"}]]></dynamic-content>

Do you see any difference? Well you could notice that there is no path of the file now. Instead of that we have JSON with few informations about our image. What's the big deal you might ask? Well the thing is that common practise in 7.0 was to fetch the path of the image in following way:

final com.liferay.portal.kernel.xml.Document document = SAXReaderUtil.read(article.getContentByLocale(locale.toString()));
final String imageUrl = document.valueOf("//dynamic-element[@name='Image']/dynamic-content");

If you used the same in 7.2 you'd end up with fetching JSON instead of the image path so if you use that as your img source it will look like that:

Screen showing the JSON inside img tag

In order not to let it happen you need to adjust your code for example in following way:

final com.liferay.portal.kernel.xml.Document document = SAXReaderUtil.read(article.getContentByLocale(locale.toString()));  
final String imageJsonString = document.valueOf("//dynamic-element[@name='Image']/dynamic-content");  
JSONObject imageJsonObject = JSONFactoryUtil.createJSONObject(imageJsonString);  
FileEntry fileEntry = PortletFileRepositoryUtil.getPortletFileEntry(imageJsonObject.getLong("fileEntryId"));  
String imageUrl = PortletFileRepositoryUtil.getDownloadPortletFileEntryURL(serviceContext.getThemeDisplay(),  
  fileEntry, StringPool.BLANK);

Please note you could have similiar situation in your FTL template but it should be easy for you to translate above Java code to FTL one.

It's worth to read that blog's entry in Liferay forum where you can read that this change was made because the Liferay developers thought not one uses this path. Unfortunately there's no better alternative (or wasn't back then). The Liferay developers didn't provide neat API you could use to fetch the Web Content's image so the above code was used - it was the solution you could found on Liferay's forum for example.

Anyway just keep that thing in mind as the images will just stop working - no error or anything in your IDE.

The fields in Search API change

In Liferay 7.2 (or 7.1) there was also a change in the naming of fields in Search API. The change was that in 7.0 the field name in

com.liferay.portal.kernel.search.Document

Looked like this:

ddm__keyword__328630__PublicationDate_en_US_String_sortable

In 7.2 on the other hand it looks like that:

ddm__keyword__328630__PublicationDate_String_sortable

See the difference? Well the _en_US part is gone. Of course it makes sense - we don't translate the PublicationDate field so why would we want to add the language key. Unfortunately you might've had a code that used the Search API and you used the name of the fields directly. You can look for such code by looking for "IndexSearchHelper", "Document" or "SearchContext" keywords and the example of such code could look like that:

final BooleanQuery publicationDateQuery = new BooleanQueryImpl();  
publicationDateQuery.addRangeTerm(String.format(PUBLICATION_DATE_TERM_FIELD_PATTERN, newsDescriptionStructureId),  
  sdf.format(from), sdf.format(to));  
indexSearcherHelper.search(searchContext, publicationDateQuery).getDocs()

Where our pattern is

private static final String PUBLICATION_DATE_TERM_FIELD_PATTERN = "ddm__keyword__%s__PublicationDate_%s";

So we can search by the field name (see addRangeTerm) and pass the field name with the language key part. The code will still compile but the results will be incorrect because we will search based on non existing field name as it was changed.

I guess this issue might be quite rare as I believe Search API is not that common but if you use that then you might have hard time finding why it stopped working.

View permission in Web Contents

Another change I couldn't find in breaking changes.

Image showing the permissions for Web Content

When you have a Web Content with permissions like on the screen above our Liferay portal will behave differently based on version. This is is connected to one of the Liferay settings available under (in 7.2):

Control panel → configuration → system settings → web content

The config is called "Article view permissions check enabled" and with it you can configure either the portal should verify the "View" permission for Web Contents or not. So:

If the the option is off - every one can see our Web Content even though the permissions (look on the screen above) suggest that only Owner should have that access.
If the option is on - there is a verification and for example Guest doesn't have access to our Web Content

The thing is the default value changed between versions. In 7.0 it was disabled by default and in 7.2 it's enabled by default.

So it's a suprise waiting for us - before everyone could see our Web Contents and now no one can. Of course if we defined the permissions then we will be fine but what if we did defined permissions for all web contents but one or two? Well that might lead to some page not working properly and since it's only one page for example then it can be easily overlooked.

Changed way of fetching current URL in FTL

In 7.0.x we could fetch current URL in FTL files using following code:

url = request.attributes['CURRENT_URL']

But after upgrading our Liferay Portal it won't work any longer. Instead we can use:

url = portalUtil.getCurrentURL(request)

it's a minor thing but undocumented one.

Is that all?

Well no. The above issues are just few examples which we might find. But still the issues especially with Search API and fetching the images are the perfect examples showing the unexpected issues that might happen to us during migration process. For contrast I also added two issues that are much easier to find just to show you that not all issues are that hidden (well the Web Content issue might be hard to find in some rare scenarios).

I hope this post will help you at least a little bit during your migration

Best regards!

Why migrating Liferay projects to a newer version is not easy

Wed, 21 Oct 2020 00:00:00 GMT

Hi!

In today's post, I would like to talk about the migration of Liferay projects between the versions, and more precisely to present the problems related to this based on my own experience. I personally participated in the migration of several projects - both from version 6.2 to 7.x and from version 7.0 to 7.2 (here additionally from CE version to DXP version). In addition, these migrations covered the whole spectrum of "cases" in terms of size - these were small modules with literally several portlets, medium projects with several database entities and several dozen portlets, but there was also one project that contained far more than 100 portlets and database entities.

It is on the basis of these experiences that I would like to show some examples of what problems we may encounter during migration. By doing so, I would like to make you aware of the fact not to believe that you will automatically manage to migrate the project, because that will most probably not be the case. I will say more - I had to debug the Liferay code myself in some cases because unfortunately, in many cases neither the documentation, nor the existing topics in the forum, nor even asking my own question, helped. Of course, if you have DXP versions, there is a chance that you will receive official support, although I know from experience that a small part of portals is based on a paid version which, due to the cost, is quite understandable.

I warn you straight away that there will not be too much code or real examples in this post because it would just take too much time to recreate them, and the production code at which my problems occurred for obvious reasons I cannot show. Nevertheless, I hope you will be able to get something useful out of the few minutes you spend reading this text.

Attention: During the writing process, it turned out to be so much material that I decided to split it into two parts. The second part will be coming soon - I simply thought, when writing this post, which was supposed to be a whole, that there was too much of it, and I did not even get to the 'main' part that I wanted to write about, namely the migration of modules.

Note! There is also second part of the article available showing some issues in code migration process. Check this out Why migrating Liferay projects to a newer version is not easy v2

Why update Liferay versions at all?

It is worth asking ourselves first - why do we migrate at all? There are a few answers - first and foremost safety. Just look at the so-called "known vulnerabilities" to see that old versions of Liferay contain a lot of shortcomings.

Another issue may be the novelties that offer newer versions of the technology used. In the case of Liferay, it is perfectly visible on the example of version 6.2 and 7.x (even 7.0) - the leap was huge due to the transition to OSGI, for example.

How does migration in Liferay look like

As I wrote earlier - this entry will not be an instruction to carry out the migration because for it is the official documentation. I'm not hiding the fact that it could probably be presented much better than in the official documentation, but it would be a topic for the whole course rather than a blog post.

However, if at this point you are not interested in going through the whole documentation, you can still assume that migration comes down to a few points in a nutshell:

Preparation of our portal - all backups, removal of duplicates or unused articles etc.
Updating the database - using a script
Migration of our modules to new versions - in case of migration from 6.2 you will most probably also need to break down the application into any modules
Tests, tests and more tests

And I'll focus on the first three to describe what can happen to us.

Preparation of a portal

The preparation of a portal is a step that is recommended in the official documentation, although, does it seems to be possible to skip it? Well, unfortunately, as it turns out, that not always it is possible. As a standard, this preparation of a portal is aimed at saving space and speeding up the migration process, although there is also a second bottom - Liferay does not always deal properly with old references. What is the old reference? For example, it could be some old webcontent that we no longer use and have long forgotten about. At one point, someone deleted the photo that this WebContent was using. It seems that nothing bad can happen because this is standard behavior - after all, nobody analyzes the content of all articles before removing the photo and really nothing bad happens in this case.

However, we move on to the migration of our portal to a higher version, we have migrated everything, the portal works until one time we download the whole LAR from the server in order to locally restore the portal and... we finish with the so-called "infinite loop". Sounds incredible? And yet, that's what happened to me. I'll add that the portal was originally 7.0.6 CE and migrated to 7.2.1 DXP. You can see the whole story (along with the advice I received) on the Liferay forum. I'll just add that in addition to the solutions proposed there with logs (which helped a lot!) I had to debug the Liferay code.

And it was only about the removed photo... In the defense of Liferay, I'll only admit that this case was specific - not always the removed photo will spoil anything (rather never), but from what I've noticed only when we upload the photo directly to the content from the editor, skipping documents & media. Maybe there must be some additional cases because I had a lot of pictures in this migrated portal (thousands), and only one of them turned out to be problematic.

How to look for such “bad” pictures? Generally, I didn't know that such a problem exists and so, as I mentioned, I ended up debugging the Liferay code along with reading logs on the DEBUG level (a lot of it) and so I found my broken WebContent. However, if I had more of such cases, most probably I would try some tool for searching "Broken references" - generally it would be to search our site for places where we have broken photos (you know - such a small icon of a photo that didn't load because of a bad source). Unfortunately, it won't work if our webcontent is not embedded anywhere - then you could probably write some small program to look for it.

Updating the database

As far as updating the database itself is concerned, I admit it's not bad. The creators of the so-called "upgrade tool" did a good job because I didn't encounter any major problems, despite migrating really large databases (one with about 8 virtual instances and the other even with about 20). In fact, I encountered only two problems strictly related to the migration of the database, but they were not tragic - considering how much it could have broken down in WebContents for example

JAX-RS configuration entries

In one of the migrated applications (7.0.6 -> 7.2.1 DXP), where the JAX-RS was used (about which in the context of Liferay you can read here) there was an error at the start of the application and it looked like this:

Screenshot of an error when launching a portal with an incorrect entry in the Configuration_ table related to JAX-RS

It was associated with incorrect entries in the Configuration_ table. They looked more or less like this:

liferay.portal.remote.rest.extender.configuration.RestExtenderConfiguration.e7c762dd-9fed-4956-8bbf-a89ec9e3a1b8
liferay.portal.remote.rest.extender.configuration.RestExtenderConfiguration.factory

There were more of them (one for "Factory" and one for each registered endpoint). Numbers will also be different (in case of the first one), so the easiest way to find them is with a query:

select * from Configuration_ WHERE configurationId like ‘%com.liferay.portal.remote.rest.extender.configuration.RestExtenderConfiguratio%’

Solution? Just delete these entries and use them during the database migration. When we migrate our JAX-RS related modules and run them all will work properly.

Release_ table

The second problem I encountered was incorrect entries in the Release_ table, which stores information about application versions or modules. The problem arose when migrating from version 6.2 GA2 to version 7.3 GA1 - the release table did not migrate correctly, although it was enough to compare its contents with the "fresh" database created in 7.3.1 and "swap" the entries and everything worked correctly.

In conclusion

As I mentioned in the introduction, this is the first of two articles on migration problems. In the second post I'll describe the topic of module migration where there are many more problems - there I'm sure I won't even describe everyone I've met because there are a lot of them, although many of them are rather due to poor documentation (and deficiencies in the so-called "Breaking changes"), but most of them are not so difficult to fix.

And finally, good luck!

If you are looking for help with such migrations in your company, do not hesitate to contact me. I can help with small migrations as well as large ones.

Continuous integration and quality inspection for Liferay 7

Fri, 26 Jun 2020 00:00:00 GMT

Hello! Today I would like to show you how to use Jenkins for continuous integration (CI) and continuous code quality inspection using SonarQube.

I decided to write this post because some time ago I had to make such process for project I work on and I couldn't find any help regarding Liferay and I had some problems with that, especially with setting SonarQube properly for Liferay workspace

Prerequisites

First we need few things:

Liferay Project - for this case I created a simple project which you can also use. It contains just a few classes and modules but is more than enough to show the basic concept. You can find it on my github
Jenkins
SonarQube

Jenkins

Lets start with Jenkins. In case you never heard of Jenkins it's an open source automation server. It's often used with code-related automation process such as building, running tests and deploying application but you can as well use it for other things such as running some application periodically for example.

To install Jenkins you should check this installing instruction site. You can install it either on your machine or using Docker. I suggest you the latter as I'm Docker fan myself but the instructions are the same either way so pick yourself :)

SonarQube

The second tool we need is SonarQube which is an open source tool for performing automatic code quality checks. It can detect code smells, bugs, duplicated code and things like that. I'll show you SonarQube later on when we'll start configuring things. The instructions for installating the SonarQube are on its official site. Again you can pick between Docker or standard installation way.

Integration

Now when we have fulfilled requirements we can start the configuration of integration between our Jenkins, SonarQube and our project.

Basically what we want to do now:

Configure the project in SonarQube
Add SonarQube Scanner plugin to Jenkins
Configure the SonarQube scanner plugin
Create a new Jenkins project for building our project and scanning with SonarQube

Configure the project in SonarQube

Before we start an integration we need to create SonarQube project. To do that we go to our SonarQube server (if installed locally it's most likely under localhost:9000), we log in (default credentials are admin/admin) and click "Add" button in top right corner Then we have to generate our token - we simply enter some name and click "Generate". The token is generated for us and we can use it from other application (such as Jenkins) to connect to SonarQube.

Generated token in SonarQube

We save the token and the project name as we need that later. If you click continue you can see the example of how to use SonarQube from Gradle or Maven but we won't need that.

Add SonarQube scanner plugin to Jenkins and configure it

In order to integrate with SonarQube we need a "SonarQube Scanner" plugin. We can install it by going to (from dashboard):

Manage Jenkins -> Manage plugins -> Available -> Search for "SonarQube Scanner" -> check the first one and click "Install with restart"

Then, after we install the plugin, we go to (from dashboard):

Manage Jenkins -> Global tool Configuration -> Under "SonarQube Scanner" we click "Add SonarQube Scanner", name it somehow and just leave the other, default options. We then click "Save"

Configuration of SonarQube scanner in Jenkins

After we add SonarQube Scanner we also need to configure our SonarQube server. To do that we go to (from dashboard):

Manage Jenkins -> Configure System -> We look for "SonarQube servers" and we click "Add SonarQube". We name it, provide URL

Create Jenkins project (task)

In Jenkins we have several types of projects. What we want is the basic "Freestyle project" which basically a type of task where we can put different stuff together (like build, test, deploy etc). We create it by just using "New item" from dashboard and selecting "Freestyle project", giving it a name and just click ok.

Jenkins new project button

New Jenkins project options

After we click ok, a newly created task setting site should appear. We won't go through all of the settings because there are many of them and most of them are for quite specific use cases. Fortunately we can click on question mark next to each field and a see a help message for the field.

When you create a new Jenkins task you should first define what exactly you want to do. In our case we want to:

Pull code from remote repository - this will be git in my example but you can also other like SVN
When we have our code we want to build it using gradle
Then we scan our code with SonarQube

In this example that's all. Usually you also want to run tests or go even further and automatically deploy the application to your server (Continious deployment) to have full CI/CD process but for now we'll just stick with continiuous builds and code check

Pull code from repository

To automatically pull code from our repository we go to "Source Code Management" section. Then we pick the system we want to use. In my case this will be Git. We also need to provide credentials - we can do that by clicking add (assuming we're doing that for the first time) and selecting "Jenkins". In new panel we provide username and password and click add. The credentials will be saved and we'll be able to reuse them with other tasks. When we finish it should like on the screen below:

Image showing the filled credentials fields

Build our application

Now we go to "Build" section. In this section we can add one or more steps for our task. The step can be for example running Gradle, invoke Ant, run shell (or windows cmd) command etc. In our case we pick "Invoke gradle". We can either choose a version we want to use (these can be configured in Jenkins settings) or use Gradle Wrapper. Using gradle wrapper is recommended way of using gradle so we'll pick that option. We check the "Make gradlew executable" (it basically does something like chmod +x ./gradlew) and in task section we put our build task

clean build

Running gradle in Jenkins configuration

Scan our code with SonarQube

We have our code built. Assuming it builds properly we want to call the SonarQube now. To do that we add another build step by clicking "Add build step" just under our gradle step. We then pick "Execute SonarQube Scanner". In "Analysis properties" we need to provide informations for SonarQube scanner so it knows how to scan our project. There are many settings we can use and you can find them all in Sonar documentation but the properties we're going to need are:

sonar.login - token you have generated in SonarQube (see "Configure the project in SonarQube" section)
sonar.projectKey - projectKey you entered when you created your project
sonar.java.binaries - you specifiy location of your project binaries
sonar.java.libraries - if they are extra libraries you should put their location so Sonar can use them while scanning
sonar.java.source - source version of your project

Check the screen below for the default values - they should work also for you unless you have some specific paths or requirements

SonarQube scanner configuration in Jenkins project

Lets test it!

Run Jenkins build we created

Finally we have it all to run our task for the first time. After we save our Jenkins task we can run it by clicking "Build now" (if we're on dashboard we have to enter the task by clicking on its name first).

We wait for a little bit. After all we should see a task number with blue dot under build history indicating that everything went ok:

Build history overview

In build history you can check see your previous builds (you can see mine is already #4 - previous 3 failed but hopefully if you follow the tutorial it will work for you at the first time! ;) ). You can also click on the number to go into specific build history entry where you can for example check console output (really useful if your task fails!).

Check the result in SonarQube

Lets go to our SonarQube project and see the result. If you're on SonarQube main page you can go to "Projects" and enter the project you've created by clicking on its name.

Note**:** Sometimes you might need to wait a while. Jenkins task sends the data to SonarQube and finishes but SonarQube need a while to process the data. For big projects there is usally a small delay between Jenkins task finishes and the results are available.

When you enter your project you can see summary for your project:

Summary of errors on SonarQube

There are tabs:

New code - it shows summary only for new code
Overall code - summary for whole codebase

The difference is that if you have an error that exists for a year then it will only be counted in "Overall code" but not in new code. That helps especially when you start scanning an existing code - usually you have a lot of bugs already and you don't want to fix them right away so you can just start by making sure that new code is good enough.

Another useful tab is "Issues". It shows the issues in the code - probably the most important tab. You can check every bug Sonar finds in your code. You can also filter by severity, type, date and many other things. What is also cool about this is that you can assign an issue to someone so from Sonar you can manage the people assigned to certain errors. You might also change type, severity or add a comment. What is also important is that you can also mark issue as fixed, false positive or won't fix - this is really helpful in some scenarios since SonarQube is just a tool and makes mistakes.

Problem with Liferay service builder code

If you looked through your SonarQube issues you probably noticed one problem - there are a lot of bugs. Is your code that bad? Well maybe but there is also one more explanation - auto generated code from service builder. In my case my test project has only three classes where I put code any yet I have almost 100 errors which is quite a lot (even though I made some mistake on purpose for this demo). And that's just from one entity in my Service Builder.

We could of course mark these bug as "false positive" or "won't fix" but lets imagine a real project - number of these bugs would be just too high. I myself worked on a project which had around 150 entities in service builder and probably your project might have much more.

Fortunately there is another way. We can tell Jenkins to ignore some paths from scanning with sonar.exclusions property. With this property you can select the paths you want to ignore. This seems easy but there are actually two problems:

If you use Liferay 7 you probably have well modularized project (or you should have!) with OSGI. That means you can't simply put one path for your services because you might have as well have 15 modules with services
We have auto generated code in *-api but also in *-service. You probably could ignore whole *-api modules in most cases but in *-service there is some of the auto generated code and some of your code (in *impl classes)

And I can think of two solutions:

For small projects, perhaps you don't have that much modules - you could simply point to proper modules and for -service to directories in that module that should be ignored
Think about some "general" rule which would apply in your case

General exclusion rule

I picked the second option for this blog post but also in my real life project so I can say it works quite well. The property is:

sonar.exclusions=**/build/resources/main/META-INF/resources/**/*.jsp,**/*-service/**/model/impl/**/*.java,**/*-service/**/service/persistence/**/**/*.java,**/*-service/**/service/base/**/*.java,**/*-service/**/service/http/**/*.java,**/*-service/**/service/util/**/*.java,**/*-api/**/dto/**/*.java,**/*-api/**/exception/**/*.java,**/*-api/**/model/**/*.java,**/*-api/**/service/**/*.java

As you can see you can separate the entries with comma. You can also use:

* - to ignore on or more character in file name
** - to ignore one or more directories

So this property basically ignores auto generated classes in both -service and -api modules of service builder. There is one problem though. This property makes some assumptions about naming in your project. For example it ignores **/*-service/**/service/util/**/* but what if you had your own module called for example blogs-service and inside you had a directory service/util? Well then this would be ignored then.

Unfortunately in this case you have to tune the general rule or just pick the option "a" and point to directories you want to exclude yourself. In most cases though that should work for you "as is".

Another test

When you put this extra property it should look much better for you. In my case out of 100 errors I have only 10 left which is great. And these ten are the real bugs or rather code smells I have in my code. But what is also important - the one error in *LocalServiceImpl is still there so we didn't accidentally ignored that class with property we added.

Errors after applying the general rule

Summary

To sum up - we have created a Jenkins job that runs the build of our project and sends the result to SonarQube. On SonarQube page we can see the result of our code quality check i.e. issues in our codebase such as security hotspots, vulnerabilities, code smells and others.

In this tutorial I didn't touch the topic of build triggers (see "Build Triggers" section in Jenkins task configuration) but you probably want one of these. This allows you to automatically run the task whenever something change in your GIT (or other) repository. There are different ways of doing that but you can read a lot regarding this topic on other sites or blogs so I left. You can also check other options in task configuration but like I said before - most of them have some specific use cases and you'll probably not need most of them at least for now.

The next step in our automation process could be automatic unit tests running or automatic deployment of your application to the server. You could also send an email for failed builds (see post-build actions in task configuration) or perhaps do something else specific to your company process.

With Jenkins you can do quite a lot so I encourage you to try it out and see what options it can give you. Keep in mind that there are also plugins that can help you integrate with different stuff you might want to use.

Hope you found that useful and see you. And remember - if you had any questions you can always write a comment or contact me.

Rafał Pydyniak - Liferay Expert Personal Blog

Fixing Liferay GA132 Bug: Client Extensions Being Removed After Modification

The Problem: Disappearing Client Extensions in Liferay GA132

The Impact: Development Frustration and Production Risks

Official Bug Status

Solutions: Creating a Custom Implementation

Option 1: Create a Custom ServiceWrapper

Option 2: Replace the CETConfigurationFactory (Recommended)

Implementation Guide: Replacing the CETConfigurationFactory

Step 1: Create a Custom CETConfigurationFactory

Step 2: Disable the Original Factory

But What About Cleanup?

Conclusion

Resolving Mockito Errors After JDK Upgrades in Liferay Portal

The Problem: Mockito Errors After JDK Upgrade

Understanding the Root Cause

The Solution: Adding JVM Arguments

For Gradle-based Liferay Projects

For Maven-based Liferay Projects

How These Arguments Work

Is it new issue?

Additional Considerations

Conclusion

Liferay: Factory Configurations

Custom configurations in Liferay

Factory configurations

Accessing factory configurations in code

Summary

Using Navigation Entry for custom functionality in Liferay core views

Background

What are OSGi Fragments

Why should you avoid using fragments for JSP overrides and why they are an issue while upgrading your Liferay version

Why would you even use OSGi fragments for JSP overrides in real life projects

Navigation Entries

How and when Navigation Entries can be used as OSGi fragment replacement

Going through the code

Adding Web-ContextPath

Navigation category and navigation entry

Custom JSP

MVC Action Command

Results

Summary

Liferay: Why am I getting "Incorrect string value" error

Database charset vs database collation

Examples of charsets and collations

Different "levels" for charset and collation settings

Solution to "Incorrect string value" issue

Useful MySQL Commands related with charsets and collations

Final words

Is Liferay Commerce production ready? Personal Experience

Few words about my experience with Liferay Commerce and why did I even create this post?

Pros and cons of Liferay Commerce

Pros: Variety of options

Pros: Documentation

Pros: Extending options

Pros: Liferay huge ecosystem

Cons: Some issues / poor design still happens

Cons: Responsiveness..

Cons: Community

Cons: It's still Liferay - most likely not for small business

Summary

Using Custom System Checkers for finding unresolved bundles

Some introduction

But hey - developer would notice that the fragment has not started

Doesn't Liferay give you some error on startup if the bundle couldn't be resolved?

Then how can we find out that the fragment has not resolved?

System Checkers

What system checkers are

How we can implement a custom checker - example of creating one which finds installed (but not resolved) bundles

Creating a OSGi command

Testing

Final thoughts

Overriding Liferay Core Permissions

Why would you want to override Liferay core permissions?

Things to consider when overriding permissions - when you should not do it?

In most cases you should not remove actions or if you really want to do that - be really careful with that

Adding permission does not mean checking for permissions. In permissions *.xml files you only define what is available in Liferay

OSGi vs non OSGi modules - two different approaches for overriding permissions

Override definition of permissions in OSGi module

Create a fragment module