Navigation :

Load Failure Response

Response when requested COB Date folder does not exist

The server would return a HTTP 500 error, and an appropriate message indicating the directory location for the specified topic and CobDate does not exist.

Reponse with Parsing Exceptions

Note: The status below indicates the status of the current request. To check if there were any errors in the data load, check the eventType in the list of “events”.

{
  "status": "success",
  "data": {
    "taskName": "LoadDataTxControllerTask-1",
    "timeTakenMs": 45957500,
    "events": [
      {
        "threadName": "csv-worker-CsvSource-Bulk-2",
        "threadId": 65,
        "clazz": "com.qfs.msg.csv.impl.Parser",
        "method": "onTranslationFailure",
        "lineNumber": 477,
        "timeStamp": 1584052234702,
        "taskName": "LoadDataTxControllerTask-1",
        "cause": {
          "type": "com.quartetfs.fwk.QuartetRuntimeException",
          "message": "Exception while translating column ProductPrice with parser Plugin Value (key=double, class com.quartetfs.fwk.format.impl.DoubleParser) in file FileSystemFileInfo [fullName=C:\\source\\tools\\data-connectors\\data-connectors-csv\\target\\test-classes\\data-samples\\data\\2019-01-01\\badproduct.csv, lastModifiedTime=Thu Mar 12 18:30:15 EDT 2020]. Parsed text was \"abc\"",
          "stackTrace": [
            {
              "methodName": "compute",
              "fileName": "CSVColumnParser.java",
              "lineNumber": 165,
              "className": "com.qfs.msg.csv.impl.CSVColumnParser",
              "nativeMethod": false
            },
            {
              "methodName": "processColumn",
              "fileName": "ColumnCalculationContext.java",
              "lineNumber": 159,
              "className": "com.qfs.msg.impl.ColumnCalculationContext",
              "nativeMethod": false
            },
            {
              "methodName": "process",
              "fileName": "ColumnCalculationContext.java",
              "lineNumber": 113,
              "className": "com.qfs.msg.impl.ColumnCalculationContext",
              "nativeMethod": false
            },                
            .
            .
            .
          ],
          "cause": {
            "type": "java.lang.NumberFormatException",
            "message": "Digit or '.' required",
            "stackTrace": [
              {
                "methodName": "parseDouble",
                "fileName": "TypeFormat.java",
                "lineNumber": 491,
                "className": "javolution.text.TypeFormat",
                "nativeMethod": false
              },
              {
                "methodName": "parseDouble",
                "fileName": "TypeFormat.java",
                "lineNumber": 562,
                "className": "javolution.text.TypeFormat",
                "nativeMethod": false
              },
              {
                "methodName": "getParsedDouble",
                "fileName": "DoubleParser.java",
                "lineNumber": 54,
                "className": "com.quartetfs.fwk.format.impl.DoubleParser",
                "nativeMethod": false
              },                 
              .
              .
              .
            ],
            "cause": null
          }
        },
        "upTime": 17315,
        "file": "C:\\source\\tools\\data-connectors\\data-connectors-csv\\target\\test-classes\\data-samples\\data\\2019-01-01\\badproduct.csv",
        "line": "[soap_21, GreenSoap, Green, abc, 2019-01-01]",
        "fileLineNumber": 18,
        "tags": [
          "health",
          "dlc",
          "source",
          "parsing"
        ],
        "eventType": "PARSING_EXCEPTION"
      },          
      {
        "threadName": "csv-worker-CsvSource-Bulk-2",
        "threadId": 65,
        "clazz": "com.qfs.msg.csv.impl.PublicationQueue$1",
        "method": "onCompletion",
        "lineNumber": 159,
        "timeStamp": 1584052234708,
        "taskName": "LoadDataTxControllerTask-1",
        "level": {
          "name": "INFO",
          "resourceBundleName": "sun.util.logging.resources.logging",
          "localizedName": "INFO"
        },
        "upTime": 17321,
        "file": "C:\\source\\tools\\data-connectors\\data-connectors-csv\\target\\test-classes\\data-samples\\data\\2019-01-01\\badproduct.csv",
        "parsingInfo": {
          "publishedObjectCount": 31,
          "characterCount": 1390,
          "lineCount": 34,
          "byteCount": 1392,
          "elapsedTime": "PT0.01S",
          "aggregatedTime": 9051401,
          "aggregatedReadingTime": 565000,
          "aggregatedDecodingTime": 224301,
          "aggregatedParsingTime": 8257599,
          "aggregatedPublicationTime": 4501,
          "exceptions": [],
          "creationTime": "2020-03-12T22:30:34.698Z",
          "numberSkippedLines": 1
        },
        "fileLineNumber": 0,
        "tags": [
          "parsing",
          "csv",
          "source"
        ],
        "eventType": "LOAD_SUMMARY"
      },
      {
        "threadId": 0,
        "lineNumber": 0,
        "timeStamp": 1584052234713,
        "taskName": "LoadDataTxControllerTask-1",
        "level": {
          "name": "INFO",
          "resourceBundleName": "sun.util.logging.resources.logging",
          "localizedName": "INFO"
        },
        "upTime": 0,
        "message": "[LOAD_SUMMARY][CSV_SOURCE] CsvSource-Bulk completed fetch batch CsvSource-Bulk-1 in 15ms ( 1 channels { StoreMessageChannel [topic=BadProducts] } and scope {CobDate=2019-01-01} ). 1 KiB 368 bytes (1392) parsed into 34 lines (31 records published)). Average throughput: 87 KiB 400 bytes (89488)/s, 2,185 lines parsed/s, 1,992 published records/s.",
        "tags": [
          "summary",
          "health",
          "dlc",
          "source"
        ],
        "eventType": "SUMMARY"
      }
    ],
    "dlcLoadingStatus": "FAILED"
  }
}

In this example, two CSV rows with errors resulted in two error events. Look at the event with type “PARSING_EXCEPTION”.

The details to look for in such exception events are file, line, fileLineNumber, eventType, detailMessage, etc.

Here’s an excerpt from one of the error messages:

"file": "C:\\source\\tools\\data-connectors\\data-connectors-csv\\target\\test-classes\\data-samples\\data\\2019-01-01\\badproduct.csv",

"line": "[soap_21, GreenSoap, Green, abc, 2019-01-01]",
"fileLineNumber": 18,
"message": "Exception while translating column ProductPrice with parser Plugin Value (key=double, class com.quartetfs.fwk.format.impl.DoubleParser) in file FileSystemFileInfo [fullName=C:\\source\\tools\\data-connectors\\data-connectors-csv\\target\\test-classes\\data-samples\\data\\2019-01-01\\badproduct.csv, lastModifiedTime=Thu Mar 12 18:30:15 EDT 2020]. Parsed text was \"abc\"",
"eventType": "PARSING_EXCEPTION"

The load summary event shows the number of rows that have been loaded:

{    
    "threadName": "csv-worker-CsvSource-Bulk-2",
    "threadId": 65,
    "clazz": "com.qfs.msg.csv.impl.PublicationQueue$1",
    "method": "onCompletion",
    "lineNumber": 159,
    "timeStamp": 1584052234708,
    "taskName": "LoadDataTxControllerTask-1",
    "level": {
        "name": "INFO",
        "resourceBundleName": "sun.util.logging.resources.logging",
        "localizedName": "INFO"
    },
    "upTime": 17321,
    "file": "C:\\source\\tools\\data-connectors\\data-connectors-csv\\target\\test-classes\\data-samples\\data\\2019-01-01\\badproduct.csv",
    "parsingInfo": {
        "publishedObjectCount": 31,
        "characterCount": 1390,
        "lineCount": 34,
        "byteCount": 1392,
        "elapsedTime": "PT0.01S",
        "aggregatedTime": 9051401,
        "aggregatedReadingTime": 565000,
        "aggregatedDecodingTime": 224301,
        "aggregatedParsingTime": 8257599,
        "aggregatedPublicationTime": 4501,
        "exceptions": [],
        "creationTime": "2020-03-12T22:30:34.698Z",
        "numberSkippedLines": 1
    },
    "fileLineNumber": 0,
    "tags": [
        "parsing",
        "csv",
        "source"
    ],
    "eventType": "LOAD_SUMMARY"    
}

The key is to check that publishedObjectCount = (lineCount - nbSkippedLines) in the LOAD_SUMMARY event.