Question

Problem with POST request to elasticsearch endpoint (unknown parameter [_type])

  • 15 June 2023
  • 4 replies
  • 352 views
vmafonso
Rank

I’m working with a legacy version of datacollector here v.3.22 and an elastic cluster (7.10.2). I’m getting the following error message:

Record1-Error ELASTICSEARCH_17 - Could not index '10' records: POST https://<ingest_route>:<port>/_bulk: HTTP/1.1 400 Bad Request {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"}],"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"},"status":400}

 

I have tried removing the @metadata/type field with a field remover and tried renaming it but I get the same error message. 

I also noticed the stage lib I’m using is for elasticsearch 7.9

Is this version of streamsets I’m working with just too outdated and can no longer support more recent version of elasticsearch? Is there any workaround I could try to solve this?

 

 

4 replies

AkshayJadhav
Rank
Userlevel 4
Badge

Thank you @vmafonso  for reaching out to the StreamSets Community. would you please share the configuration of the elasticsearch stage.

vmafonso
Rank

Hi @AkshayJadhav thank you for the quick reply. 

Here’s the current config:

 

General:

Name: Elasticsearch 1

Description:

Stage Library: Elasticsearch 7.9.0 

Required Fields:

Preconditions: 

On Record Error: Send to Error 

 

Elasticsearch:

HTTP URLs: <ingestion_route>

HTTP Port: 9200

Use Security: YES

Index: <index_name>-${YYYY()}-${MM()}-${DD()}

Mapping: <index_name>

Document ID:

Additional HTTP Params:

Detect Additional Nodes in Cluster: NO

Time Basis: ${time:now()}

Data Time Zone: + 00:00 UTC (UTC)

Parent ID: 

Routing:

Data Charset: UTF-8

Default Operation: INDEX

Unsupported Operation Handling: Discard

Additional Properties: {}

 

Security:

Mode: Basic

User Name: <username>

Password: <password>

Enable SSL: YES

SSL TrustStore Path: <path_to_truststore>

SSL TrustStore Password: <truststore_password>

 

 

 

AkshayJadhav
Rank
Userlevel 4
Badge

I have checked with the latest version and I do not see any issues with connectivity. 

vmafonso
Rank

You’ve checked with the latest version of elasticsearch? So there should be no incompatibility issues? Did you configure TLS client authentication for elastic? 

Reply


Terms & Conditions