Releases 6.1

Release 6.1.0

Close to the release of TYPO3 8 LTS we are happy to announce EXT:solr 6.1 that is compatible with TYPO3 8 LTS.

New in this release

In this release we've merged over 160 pull requests! With these pull requests several new features and bugfixes were added.

Compatible with TYPO3 8 LTS and PHP 7.1 ready

EXT:solr 6.1 is ready for TYPO3 8 LTS and php 7.1 while keeping the compatibility to TYPO3 7.6 LTS and PHP 5.5 and 5.6.

Backend Performance Improvements

During the development of version 6.1 there have been a lot of smaller and bigger performance improvements. The following changes might be interesting.

Allow to use the closest configuration in the page tree

When a record is saved in the backend by now the whole TypoScript configuration is evaluated for the page where the record is located. In many setups it is enough to just use the closest template in the rootline to parse the configuration. Since there are cases, where this method does not work (e.g. when you use conditions based on page ids) you need to switch this behaviour explicitly on, by enable "useConfigurationFromClosestTemplate".

Add caching when record monitor is used

To optimize the performance the RecordMonitor and GarbageCollector class have been splitted into several components. These components use the TYPO3 caching framework to cache result that do not need to be retrieved multiple times.

Add a global list of monitored tables

If a record is relevant can only be decided in the context of a Solr site because the TypoScript configuration defines the indexing configuration.

When you want to tweek the backend performance you can define a global list of monitoredTables. Other tables will then be ignored and also the parsing of the TypoScript configuration is not needed then.

New Index Inspector

Since extjs will be removed more and more from the TYPO3 core we decided to migrate the index inspector to fluid. You can use the index inspector as usual from "Web > Info > Search Index Inspector" to analyze which documents are in the solr server for a specific page.

Related Issues:

Allow to monitor records outside the siteroot

When the property "additionalPageIds" is configured for an indexing configuration this configuration is now taken into account in the record monitor to re-queue these elements when a change in the backend is done.

Since this is an expensive operation, you can disable this feature by configuring the extension setting (useConfigurationTrackRecordsOutsideSiteroot) in the extension manager.

Related Issues:

Use TYPO3 logging framework instead of devlog

By now the devLog from TYPO3 was used to log messages in the extension. Since several years there is a more flexible logging framework available in the TYPO3 core. In EXT:solr 6.1 we make use of this and use the TYPO3 logging framework to write the logs.

Related Issues:

Use ::class and shorten array syntax

The minimum php version (PHP 5.5) allows us to benefit from PHP language features that make the code more readable:

  • Short array syntax: We replaced the parts in the code where [] can be used instead of array()

  • ::class: By using ::class all places where classes get instanciated with GeneralUtility::makeInstance can be simplyfied

Related Issues:

Support partial matches with N-Gram and Edge N-Gram

For some search use cases it makes sence to support partial matches e.g. when you search for "boch" and want to mach a field with the content "bochum".

EXT:solr 6.1.0 shippes new dynamic field types that make "N-Gram" and "Edge N-Gram" processed fields available:

  • Edge N-Gram Singlevalue: \*_textEdgeNgramS Edge Ngram (hello => hello, hell..)

  • Edge N-Gram Multivalue: \*_textEdgeNgramM Edge Ngram (hello => hello, hell..)

  • N-Gram Singlevalue: \*_textNgramS Ngram (hello => he,ll,lo,hel,llo)

  • N-Gram Multivalue: \*_textNgramM Ngram (hello => he,ll,lo,hel,llo)

See also:

https://cuiborails.wordpress.com/2012/07/13/apache-solr-ngramedgengram/

Related Issues:

Allow to disable siteHash check by using allowedSites = *

The siteHash is unique for a Solr site in the TYPO3 system. When Solr does a query the allowedSites setting can be used to control the set of documents that are queried in a Solr core. This is useful when you want to split the content from multiple sites in a single Solr core. In some cases it is useful to disable this limitation.

E.g. when:

  • You have data in Solr that comes from another system

  • When you want to search across multiple sites

Before the extension "solr_disablesitehash" was required to turn this sitehash check off. With EXT:solr 6.1.0 we've changed the meaning of the allowedSites:

  • Before: * was the same as __all, which means all sites in the system

  • After: __all is still handled as __all sites in the system, but * now means every site (same as no check at all)

Migration: When you are using * for query.allowedSites change the setting to __all.

Related Issues:

Update jQuery version

In this release we have updated the shipped jQuery and jQueryUi components to the following versions:

  • jQuery: v3.1.1

  • jQuery UI: v1.12.1

Related Issues:

Add Solr access filter plugin 2.0.0

The Solr access filter plugin has been optimized to use a Solr post filter. By using a post filter the performance of this plugin is much better, because less documents need to be evaluated. In this release we ship this new version 2.0.0 of the access filter with the default configSet and in our docker container.

Related Issues:

Preparation for Doctrine migration

In the next release we will drop the support of TYPO3 7 LTS and only support 8 LTS. Since in the TYPO3 core the database access was migrated to Doctrine, we will use Doctrine as well.

To simplify this, we have centralized a lot of database queries in this release and also dropped duplicate queries to reduce to amount of work that needs to be done for this in the next releaese.

Related Issues:

Allow self signed ssl certificates

Not every project can use officially signed SSL certificates for all stages. Because of that we allowed for the page indexing to index a host, with a self signed certificate.

You can enable this feature by configuring the extension setting (allowSelfSignedCertificates) in the extension manager to true.

Related Issues:

Add cObject support for Solr settings

When you use EXT:solr in a deployment scenario (e.g. platform.sh) you maybe want to define the Solr endpoints by environment variables or from variables in TYPO3_CONF_VARS. Both approaches are supported by the TypoScript TEXT object. Therefore it makes sence for the Solr endpoint settings in plugin.tx_solr.solr to support the usage of cObjects there. This allows you to define connections like this:

Addition to AdditionalConfiguration.php:

$GLOBALS['TYPO3_CONF_VARS']['EXTCONF']['solr']['host'] = 'mysolrserver.de';

Usage in TypoScript:

plugin.tx_solr.solr {
    host = TEXT
    host {
        value = localhost
        override.data = global:TYPO3_CONF_VARS|EXTCONF|solr|host
    }
}

Related Issues:

Deprecated code

For the continues improvement of the codestructure and preparation for upcoming tasks, the following methods have been marked as deprecated and will be removed in EXT:solr 7.0 (next release):

  • GarbageCollector::cleanIndex

  • Query::addSortField

  • Query::getSortingFields

  • SolrService::getSchemaName

  • Util::timestampToIso

  • Util::isoToTimestamp

  • Util::timestampToUtcIso

  • Util::utcIsoToTimestamp

  • Util::getRootPageId

  • Util::isRootPage

  • Util::getSiteHashForDomain

  • Util::resolveSiteHashAllowedSites

  • Item::updateIndexedTime

  • Queue::getIndexingConfigurationByItem

  • Queue::getIndexingConfigurationsByItem

  • Queue::getItemsCountBySite

  • Queue::getRemainingItemsCountBySite

  • TypoScriptConfiguration::getIndexQueuePagesAllowedPageTypesArray

  • QueryTest::canAddSortField

  • Site::getLanguages

  • Site::getFirstAvailableSite

  • Site::getAvailableSites

  • Site::getAvailableSitesSelector

Please check your deprecation log and replace the usages in your code.

Breaking changes

In this release some breaking changes have been required. Please follow the notes below when you migrate from 6.0 to 6.1

Default variantId changed

The default variantId was table/uid before since this id is not unique across multiple TYPO3 system, a system hash was added before. The scheme of the new variantId is systemhash/table/uid and allows to use grouping when data from multiple TYPO3 systems get indexed and searched.

Migration:

Reindex the Solr data to get the new variantId.

Related Issues:

Semantic of allowedSites changed

Migration:

When you use plugin.tx_solr.search.query.allowedSites = * you should now use plugin.tx_solr.search.query.allowedSites = __all

Related Issues:

Setting debugDevlogOutput replaced by debugOutput

Because the devLog was replaced by the logging framework, the setting plugin.tx_solr.logging.debugDevlogOutput does not make sence anymore.

Migration:

When you want to see the log output in the backend or frontend please use plugin.tx_solr.logging.debugOutput now.

Related Issues:

"Only variables should be assigned by reference" in IndexQueueIndexer::preAddModifyDocuments()

Along we the removal of the reference GeneralUtility::getUserObj was replaced with GeneralUtility::makeInstance, because the usage with ":" is deprecated since TYPO3 8 and will be removed.

Migration:

When you reference custom indexer (in $GLOBALS['TYPO3_CONF_VARS']['EXTCONF']['solr']['IndexQueueIndexer']['indexItemAddDocuments']) you should reference only an autoloadable classname.

The following steps are required:

  • Move your Indexer to an own extension with proper autoloading configuration

  • Reference the autoloadable classname

Related Issues:

Bugfixes

StatisticRepository::getTopKeyWordsWithOrWithoutHits $limit, $withoutHits have no default values

https://github.com/TYPO3-Solr/ext-solr/issues/1143

Wrong TS path in frontendDataHelper documentation

https://github.com/TYPO3-Solr/ext-solr/issues/964

Invalid include paths after scheduler run

https://github.com/TYPO3-Solr/ext-solr/issues/921

Backend Summary not working

https://github.com/TYPO3-Solr/ext-solr/issues/731

Queue initialization returns wrong (duplicate) results for second site root in multisite environment

https://github.com/TYPO3-Solr/ext-solr/issues/488

solrconfig.xml More Like This Handler configuration is broken

https://github.com/TYPO3-Solr/ext-solr/issues/765

Fix position of default stopwords

https://github.com/TYPO3-Solr/ext-solr/issues/578

Bug/unwanted feature in method getPages in Site.php

https://github.com/TYPO3-Solr/ext-solr/issues/652

Custom field is not indexed for custom page queue configuration

https://github.com/TYPO3-Solr/ext-solr/issues/842

GarbageCollector fails to check endtime correctly

https://github.com/TYPO3-Solr/ext-solr/issues/1212

"Only variables should be assigned by reference" in IndexQueueIndexer::preAddModifyDocuments()

https://github.com/TYPO3-Solr/ext-solr/issues/1183

Use urlencode for deletion of synonyms and stopwords

https://github.com/TYPO3-Solr/ext-solr/issues/1205 https://github.com/TYPO3-Solr/ext-solr/issues/1206

Contributors

Like always this release would not have been possible without the help from our awesome community. Here are the contributors for this release.

(patches, comments, bug reports, reviews, ... in alphabetical order)

  • Andreas Allacher

  • Andriy Oprysko

  • Arno Schoon

  • Frank Nägler

  • Frans Saris

  • Ingo Renner

  • Josef Glatz

  • Markus Kobligk

  • Rafael Kähm

  • Rasmus Larsen

  • Sascha Egerer

  • Thomas Hohn

  • Timo Hund

  • Tomas Norre Mikkelsen

Also a big thanks to our partners that have joined the EB2017 program:

  • Amedick & Sommer

  • amarantus - media design & coding Mario Drengner & Enrico Nemack GbR

  • Animate Agentur für interaktive Medien GmbH

  • artif GmbH & Co. KG

  • AVM Computersysteme Vertriebs GmbH

  • b:dreizehn GmbH

  • Bitmotion GmbH

  • cab services ag

  • Causal Sàrl

  • Creative360

  • cron IT GmbH

  • data-graphics GmbH

  • Deutscher Ärzteverlag GmbH

  • Deutscher Volkshochschul-Verband

  • Die Medialen GmbH

  • dörler engineering services

  • hauptsache.net

  • Havas Düsseldorf GmbH

  • itl AG

  • jweiland.net

  • KEMWEB GmbH & Co. KG

  • Leibniz Universität IT Services, Hannover

  • Lime Flavour GbR

  • LOUIS INTERNET

  • Maximillian Walter

  • Mercedes-AMG GmbH

  • mpm media process management GmbH

  • n@work Internet Informationssysteme GmbH

  • Netcreators BV (Netherlands)

  • NetSpring s.r.l.

  • netz-haut GmbH

  • polar gold

  • punkt.de

  • sitegeist media solutions GmbH

  • Star Finanz GmbH

  • Studio 9 GmbH

  • stratis

  • systime

  • takomat Agentur GbR

  • Triplesense Reply

  • Typoheads GmbH

  • UEBERBIT GmbH

  • WACON Internet GmbH

  • Universität Bremen

  • webconsulting business services gmbh

  • zdreicom AG

  • zimmer7 GmbH

Thanks to everyone who helped in creating this release!

How to Get Involved

There are many ways to get involved with Apache Solr for TYPO3:

  • Submit bug reports and feature requests on GitHub

  • Ask or help or answer questions in our Slack channel

  • Provide patches through Pull Request or review and comment on existing Pull Requests

  • Go to www.typo3-solr.com or call dkd to sponsor the ongoing development of Apache Solr for TYPO3

Support us in 2017 by becoming an EB partner:

http://www.typo3-solr.com/en/contact/

or call:

+49 (0)69 - 2475218 0