EXT: mnoGoSearch¶

Search results in the Frontend¶

The following screenshot shows search results. Notice file type icon in the first result (OpenOffice document), relevancy indicator (green bar), size and last modification date. The second result did not provide last modification date and it is not displayed in the result.

The extension uses rich page browser to allow better navigation. Page browser can be customized to show as many page links as necessary:

Configuring what to index¶

The following screenshot shows Backend configuration of web space to be indexed. It says that the whole web site should be indexed:

Next, parts of the web site are prohibited from being indexed. These pages contain news and FAQ items. We will index them differently.

Finally we index FAQ items and news. Here is how indexing of news look like:

The reasons to index news, FAQ and some other records like this will be explained later in this manual.

How mnoGoSearch decides what to index¶

mnoGoSearch sees web sites as a hierarchical structure. When indexing, it needs to know where to start indexing. Typically the start of the hierarchy is the root of the web site (like :code:``http://example.com/ <http://example.com/>`_` ). But if necessary there can be many starting points (like :code:``http://example.com/products/ <http://example.com/>`_` and :code:``http://example.com/services/ <http://example.com/services/>`_` ). In this case search will be limited to corresponding starting points and everything below them (i.e. :code:``http://example.com/products/navigation/ <http://example.com/products/navigation/>`_` ) . Any pages outside of the configured starting points are not indexed and therefore not searchable.

The important point in the information above is that web site can be indexed as whole ( :code:``http://example.com/ <http://example.com/>`_` ) or as parts. When indexing as parts, site URLs should be hierarchical, which implies usage of RealURL or CoolURI.

When the whole site in indexed, some pages may still need to be excluded. mnoGoSearch provides a way to disallow certain pages from indexing. This can be accomplished by either using No search checkbox in page properties. When multiple pages starting from a certain page should not be indexed (like checkout pages), mnoGoSearch allows to disable hierarchies by specifying path to the hierarchy.

Specifying pages to index¶

To specify pages for indexing, an indexing configuration record should be created. While creating these records, it is important to keep in mind that mnoGoSearch works with URL path hierarchies.

The first step in specifying pages for indexing is to choose where indexing records are stored. Typically it will be a web site home page or a storage folder. It does not make much difference. However it is good to be consistent and keep all indexing records for a web site on a single page. It allows to see what is actually indexed and what is excluded from indexing.

To create indexing configuration record, navigate to the page and use List module to create indexing configuration record. By default records are of the type “Server”. This is a simplest possible type. It specifies indexing starting point as a path withing the web site. For example, to index the full web site, it should be http://example.com/ (assuming that example.com is your web site domain). Note the trailing slash, it is necessary if the URL does not include any other path. Below is how such indexing records look like:

Additional options include indexing period (24 hours is the default) and “Additional indexing configuration”. The latter allows to enter mnoGoSearch configuration directives directly. They will be appended to the generated indexer configuration. Information about directives can be found at http://mnogosearch.org/doc33/ . Notice that this field is not validated and any wrong directives will result in fatal error during indexing.

The next type of indexing records is a “Realm”. Realm is very similar to “Server” but it allows to use regular expressions or wildcards to specify paths. For example, one can enter http://example.com/(news\|faq)/.\* as a path. Make sure that correct comparison type is specified:

Excluding parts of the web site from indexing¶

To exclude parts of the web site from indexing, create an indexing configuration record as described above bit set method to “Disallow”. It will prohibit any pages starting from the current path from indexing. The the screenshot above (“Real” record).

Notice that such records should appear in the List module before records that en able site indexing. The first record takes precedence when matching URLs. For example, consider http://example.com/page/?excludeMe=1. This order is correct:

• Disallow: *?excludeMe=*
• Allow: http://example.com/

This will first check “disallow” rule. If it matches, it will be used. It means that http://example.com/page/ will be indexed but http://example.com/page/?excludeMe=1 will not. However consider these rules:

• Allow: http://example.com/
• Disallow: *?excludeMe=*

Now both URLs will match because they match to the first rule and “disallow” rule will never work.

So when disallowing some pages from being indexed, always put disallow rule before the rule that allows indexing.

Indexing only real content¶

To improve search relevancy some parts of the page should be excluded from indexing. Such parts include navigation (menu), logo, copyright, statistics, partner links, copyright, etc. Typically only the real content should be included into index.

Special HTML comments can be added to the page to tell the indexer what parts of the site should be indexed. There can be many such markers on a single page. Here is a HTML fragment that illustrates how to add such markers:

<body>
<div id=”logo”>My site</div>
<!-- menu -->
<ul>
       <li><a href=”/products/”>Products</a></li>
       ...
</ul>
<!-- content -->
<!--TYPO3SEARCH_begin-->
<div id=”content”>
    Here goes real web site content...
</div>
<!--TYPO3SEARCH_end-->
<ul id="partner-links">
       <li><a href="http://example.com/">Partner web site</a></li>
       ...
</ul>
<!-- extra content -->
<!--TYPO3SEARCH_begin-->
<div id=”extra-content”>
    Here goes another content block...
</div>
<!--TYPO3SEARCH_end-->
<div id=”copyright”>Copyright © My company.</div>
</body>

In the example above content inside TYPO3SEARCH_xxx will be indexed and all links outside of these comments will be followed (added to the indexer queue). Notice that there must be no spaces or line breaks in these comments. They must be spelled exactly as shown in the example above.

Note that TemplaVoila creates such markers automatically. Other templating engines do not add such markers automatically.

Indexing records¶

In certain cases indexing content as pages is not efficient. For example, it is more efficient to index news records as records than as pages. When indexing news as pages, it adds more content than necessary to the index, increase load on the web server and lowers search relevance. When indexing news items as records, mnoGoSearch indexes only title and text fields fields. Thus only true news text is searchable.

Same applies to the FAQ (extension irfaq ) and some other extensions that store information as records.

To index records, indexing configuration for them should be created. To create indexing configuration for records navigate to the page of the web site you have chosen to store indexed configuration at. Then create indexing configuration record and set its type to “Records”. Next choose the table you want to index. The form will refresh. Here is how it will look like of “News” table from tt\_news extension is chosen:

The form requires a title and text fields of the record to be selected. There must be one title field and one or more text fields to index. Text fields will be concatenated together during indexing. Notice that no conversion done on fields. Thus using “Archive date” in the form above will not be useful because this field is stored as integer value in the database. Only true text fields should be selected.

Next parameter to specify is URL parameters for the item's single view. For most extensions it looks like &tx\_extkey\_pi1[showUid]={field:uid} . For tt\_news it looks like shown on the screenshot above. The & symbol in the beginning of the parameter is mandatory. {field:uid} is replaced uid of the record. No other substitutions available.

It is possible to limit indexing to records from the certain storage folder. This way, for example, only news records of the web site will be indexed and not any imported news in another sysfolder.

Indexing files¶

Indexing files is possible in the same way as indexing pages. Specify correct path to files ( :code:``http://example.com/fileadmin/ <http://example.com/fileadmin/>`_` and :code:``http://example.com/uploads/ <http://example.com/uploads/>`_` ) to allow indexing them. The rest is done automatically. Directories must show index of files in them (use Apache mod\_autoindex ).

To index file you successfully you must ensure that file parsing applications (like catdoc or pdftotext are installed on the server to the default places, normally /usr/bin ).

Currently mnoGoSearch supports indexing for the following file types:

AddType application/vnd.oasis.opendocument.text *.sxw
AddType application/vnd.oasis.opendocument.text *.odt

Indexing large file collections¶

If number of files is large, it does not make sense to fetch them all using HTTP. In this case an additional directive into the “Additional configuration” field of the indexing configuration for files. This directive will force the indexer to access files locally instead of fetching them through HTTP. Assuming that files are located at :code:``http://example.com/fileadmin/fileserver/ <http://example.com/fileadmin/fileserver/>`_` and physically at /path/to/fileadmin/fileserver/ , the following directive should be added:

Alias http://example.com/fileadmin/fileserver/ file:///path/to/fileadmin/fileserver/

Notice the correct number of slashes in paths.

Indexing https pages¶

Indexing https pages with self–signed certificates is not possible directly. mnoGoSearch indexer will refuse to index sucvh pages because it will not see certificate as valid. If obtaining a valid certificate is not an option, there is another way to index such pages. For that an utility named “curl” should be installed on the server.

Firsts, navigate to the web site root and execute the following command:

php typo3/cli_dispatch.phpsh mnogosearch -d | grep X-TYPO3

It will produce the output similar to:

HTTPHeader "X-TYPO3-mnogosearch: d3e203fdb699f7ba6ad7396fdba5c25a"

Note the part in quotes.

Next create a new file named curl.sh somewhere in the file system. If many sites run on the same host, it makes sense to put this file inside the web site space. Put the following content into this file:

#!/bin/sh
curl -i -k -H "X-TYPO3-mnogosearch: d3e203fdb699f7ba6ad7396fdba5c25a" $1 2>/dev/null

Note the part in quotes, it is taken from the output of the previous command. Do not copy this example! The header is unique for each site, even for sites running on the same server!

-H option adds a special HTTP header to the HTTP request. This header tells the extension that indexer is running. The extension will exclude all content outside of TYPO3SEARCH_xxx markers from indexed data. See “Indexing only real content” chapter for more information.

This script will fetch https pages even if certificate is self–signed. Make this file accessible and executable for the current use only:

chmod 0700 /path/to/curl.sh

Warning! Setting permissions like this is extremely important! Neither web server should be able to read this file, nor execute it. If permissions are not set correctly, security of the web site will be compromised!

Next add the following lines to the “Additional configuration” of the first indexing configuration you have:

Alias https:// exec:/path/to/curl.sh?https://

This will call this script for https:// scheme to fetch pages. Now https pages with self–signed certificates can be indexed too.

Make sure that /path/to/curl.sh points to the script. /path/to above is the placeholder for the real path.

Using TypoScript¶

To create a short simple form (equivalent to “macina_searchbox” for indexed search) on each page do the following in TypoScript:

lib.search_form < plugin.tx_mnogosearch_pi1
lib.search_form.mode = short_form

Now lib.search_form can be used for replacing a marker or as a TemplaVoila object. You can also change other options (like template file). See the “Configuration” section later in this manual.

Notice that by default form is not cacheable. Non–cacheable form will show search terms in the search box after the submission. This may slightly decrease web site performance. To avoid this the following line can be added after the two lines shown above:

lib.search_form = USER

See also “Using HTML” below to make even better performance when using search forms.

Using page module¶

When using Page module, insert mnoGoSearch plugin to the page and select the desired form in the plugin properties:

Using HTML¶

Instead of using plugin or TypoScript it is possible to have the form directly in the web site template. The following is the minimum required form mark up for mnoGoSearch:

Note that it needs proper “action” URL. If you plan to use Google Analytics to track search results, the method must be “get”. Otherwise it can be “post”. Here is HTML:

<form action="/search/" method="get">
       <input type="text" name="tx_mnogosearch_pi1[q]" value="" />
       <input type="submit" name="tx_mnogosearch_pi1[submit]" value="Search" />
</form>

This method is recommended for better web site performance. Plugin for this extension is defined as USER_INT, which means that the plugin is never cached. Having this plugin on every page may cause a slightly lower web site performance. Notice that using form directly in HTML will show search field in such form empty after submission.

Plugin mode¶

Plugin options contain only two controls: plugin mode and optional selector to limit search to certain web space. Plugin mode can be set to one or more of:

• Short search formThis form typically contains a short search box and a submit button
• Long search formCurrently it is identical to the short search form but has a larger search box, different text on the button. In future this form will be extended to contain search limit controls.
• Search resultsThis mode shows search results

Limiting search to a certain web space¶

If mnoGoSearch SQL database contains index for several sites, search will return results for every site. Sometimes search results should be limited to the current domain only or even to a part of the domain.

To limit search results to the domain, the field named “Limit search to” should be used. If this field is empty, search is not limited.

When this field is not empty, it must contain indexing configuration records that meet two criterias:

• they are of type “Server” or “Records” (“Realm” is not supported!)
• for “Server” type record indexing method must be “Allow” (any other method is not supported)

When field is not empty, mnoGoSearch will take all URLs from this field and returns results only that match these records. For example, if this field contains the following limits:

• Server: http://example.com/products
• Records: News, pid=12345

So if the whole server is indexed (http://example.com/), search result will contain only pages with news and pages with search terms on http://example.com/products and any page below in the page hierarchy.

Notice: it is impossible to use “Disallow” records! So it is not possible to disallow “News” and allow the rest of the site.

This feature behaves very similar to Google's domain search feature. In Google it is possible to search for a term everywhere by typing “term” in the Google search box. However it is possible to limit searach results to certain domains only using “term site:example.com/products” in the Google search box.

See “Specifying web space to search” above for information about indexing configurations.

TYPO3SEARCH_xxx comments are not respected. What is wrong?¶

Here is the checklist:

• make sure that there are no spaces around TYPO3SEARCH_xxx markers. They must look exactly like <!--TYPO3SEARCH\_begin--> and <!--TYPO3SEARCH\_end-->
• check that comments are in the right order and not nested. The first comment must be TYPO3SEARCH_begin, followed by TYPO3SEARCH_end. Two consecutive TYPO3SERACH_begin or TYPO3SEARCH_end comments will mess up the indexing
• if nc\_staticfilecache extension is installed, check its version. Extension with versions < 2.3.2 conflict with mnogosearch extension. Read the bug report at http://forge.typo3.org/issues/show/2291 and follow instructions there

I experimented and messed up my index. How do I clear it?¶

Run command line tool like this:

php typo3/cli_dispatch.phpsh mnogosearch -x -Cw

Answer YES (in block letters!) to the prompt.

I removed a page. How do I remove it from index?¶

The extension does it automatically. There is no need to do anything like this. Same happens when page is renamed: old address is deleted from index and new address is added.

I receive a error “Got error 139 from the database engine” while indexing¶

This may happen if you use database mode blob and InnoDb as your default database engine. To fix the problem open mysql command line tool for the search database and run this command:

alter table bdicti engine=myisam;

This will convert corresponding table to MyISAM format. Next restart the indexer.

There seems to be a clone of mnoGoSearch called DataParkSearch. What is it?¶

This is a fork of the mnoGoSearch project. It focuses on documentation and code quality improvements. It lacks PHP extension, so it cannot be used as a replacement for mnoGoSearch in TYPO3. mnoGoSearch PHP extension is incompatible with DataParkSearch.

What does “mnoGoSearch” mean?¶

This is about playing with words. Original developers are from Russia. “Mnogo” in Russian means “much” or “many” or “a lot”.

templateFile¶

mode¶

form¶

search¶

->FORM¶

The table below lists TypoScript properties for search forms.

-> ADVANCED¶

The table below lists TypoScript properties for search forms.

-> SELECTOR¶

The table below lists TypoScript properties for search forms.

->SEARCH¶

The table below lists TypoScript properties for search results view.