GenerateCloudSearchData Wyam.AmazonWebServices

Generates bulk JSON upload data to get documents into Amazon CloudSearch.
This module creates a single document from a pipeline with JSON data containing the correctly formatted commands for Amazon CloudSearch. Note that this just creates the document. Once that document is written to the file system, you will still need to upload the document to a correctly configured CloudSearch instance using the API or Amazon CLI.

Package

This module exists in the Wyam.AmazonWebServices package which is not part of the core distribution. Add the following preprocessor directive to your configuration file to use it:
#n Wyam.AmazonWebServices

Examples

Pipelines.Add("CloudSearchData",
    Documents("NameOfAPriorPipeline"),
    GenerateCloudSearchData("Id", "Body")
       .AddField("type", "post")
       .AddField("length", d => d.Content.Count())
       .MapMetaField("title", "Title")
       .MapMetaField("tags", "Tags", o => o.Split(",".ToCharArray())),
    Meta("WritePath", "cloudsearch_data.json"),
    WriteFiles()
);

Usage

  • GenerateCloudSearchData(string idMetaKey, string bodyField)

    Generates Amazon CloudSearch JSON data.

    • idMetaKey

      The meta key representing the unique ID for this document. If NULL, the Document.Id will be used.

    • bodyField

      The field name for the document contents. If NULL, the document contents will not be written to the data.

Fluent Methods

Chain these methods together after the constructor to modify behavior.

  • AddField(string fieldName, Func<IDocument, Object> execute)

    Adds a function-based field value. The function will take in a document and return an object, which will be the field value.

    • fieldName

      The CloudSearch field name.

    • execute

      A function of signature Func<IDocument, object>. If the function returns NULL, the field will not be written.

  • AddField(string fieldName, Object fieldValue)

    Adds a literal field value.

    • fieldName

      The CloudSearch field name.

    • fieldValue

      The value.

  • MapMetaField(string fieldName, string metaKey, Func<Object, Object> transformer = null)

    Adds a mapping from meta key to CloudSearch field. When provided, the contents of the meta key will be written to the provided field name.

    • fieldName

      The CloudSearch field name.

    • metaKey

      The meta key. If the meta key does not exist, the field will not be written.

    • transformer

      An optional function that takes a string and returns an object. If specified, it will be invoked on the meta value prior to serialization. If the function returns NULL, the field will not be written.

GitHub