Using AWS S3 for File Storage

2
Hi, I'm experimenting with Mendix on Pivotal Cloud Foundry. For that I'm also using AWS S3 for persistant file storage. All seems well, it's all fairly easy to configure and get running. Only thing still blocking me is the actual file upload to AWS S3: When trying to upload I file I get the error posted below. Any tips, pointers, ideas? Update: Here are my environment variables: Message: Uploading file failed: More data read than expected: dataLength=4767; expectedLength=3901; includeSkipped=false; in.getClass()=class java.io.BufferedInputStream; markedSupported=true; marked=0; resetSinceLastMarked=false; markCount=1; resetCount=0 [User 'Anonymous_c6674ce9-4add-4d61-b879-6f6e7234b5af' with session id '66bf1848-2393-461f-9c44-8bc7af9ec465' and roles 'Administrator'] Stacktrace: com.amazonaws.AmazonClientException: More data read than expected: dataLength=4767; expectedLength=3901; includeSkipped=false; in.getClass()=class java.io.BufferedInputStream; markedSupported=true; marked=0; resetSinceLastMarked=false; markCount=1; resetCount=0 at com.amazonaws.util.LengthCheckInputStream.checkLength(LengthCheckInputStream.java:155) at com.amazonaws.util.LengthCheckInputStream.read(LengthCheckInputStream.java:110) at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:73) at com.amazonaws.services.s3.internal.MD5DigestCalculatingInputStream.read(MD5DigestCalculatingInputStream.java:98) at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:73) at com.amazonaws.event.ProgressInputStream.read(ProgressInputStream.java:151) at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:73) at com.amazonaws.auth.AwsChunkedEncodingInputStream.setUpNextChunk(AwsChunkedEncodingInputStream.java:289) at com.amazonaws.auth.AwsChunkedEncodingInputStream.read(AwsChunkedEncodingInputStream.java:155) at org.apache.http.entity.InputStreamEntity.writeTo(InputStreamEntity.java:98) at com.amazonaws.http.RepeatableInputStreamRequestEntity.writeTo(RepeatableInputStreamRequestEntity.java:153) at org.apache.http.entity.HttpEntityWrapper.writeTo(HttpEntityWrapper.java:98) at org.apache.http.impl.client.EntityEnclosingRequestWrapper$EntityWrapper.writeTo(EntityEnclosingRequestWrapper.java:108) at org.apache.http.impl.entity.EntitySerializer.serialize(EntitySerializer.java:122) at org.apache.http.impl.AbstractHttpClientConnection.sendRequestEntity(AbstractHttpClientConnection.java:271) at org.apache.http.impl.conn.ManagedClientConnectionImpl.sendRequestEntity(ManagedClientConnectionImpl.java:197) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:257) at com.amazonaws.http.protocol.SdkHttpRequestExecutor.doSendRequest(SdkHttpRequestExecutor.java:47) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:713) at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:518) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805) at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:685) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:460) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:295) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3736) at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1461) at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1331) at com.mendix.storage.s3.internal.AmazonS3StorageService.singlePartUpload(AmazonS3StorageService.scala:50) at com.mendix.storage.s3.internal.AmazonS3StorageService.store(AmazonS3StorageService.scala:31) at com.mendix.storage.s3.internal.AmazonS3StorageComponent.store(AmazonS3StorageComponent.scala:60) at com.mendix.filedocumentprocessor.internal.FileDocumentProcessorImpl.writeFile(FileDocumentProcessorImpl.java:157) at com.mendix.filedocumentprocessor.internal.FileDocumentProcessorImpl.storeFileDocument(FileDocumentProcessorImpl.java:114) at com.mendix.core.component.InternalCore.storeFileDocumentContent(InternalCore.java:1691) at com.mendix.webui.requesthandling.FileRequestHandler.doUpload(FileRequestHandler.scala:167) at com.mendix.webui.requesthandling.FileRequestHandler.handleUpload(FileRequestHandler.scala:88) at com.mendix.webui.requesthandling.FileRequestHandler.handleRequest(FileRequestHandler.scala:61) at com.mendix.webui.requesthandling.FileRequestHandler.processRequest(FileRequestHandler.scala:54) at com.mendix.externalinterface.connector.MxRuntimeConnector$1.execute(MxRuntimeConnector.java:68) at com.mendix.externalinterface.connector.MxRuntimeConnector$1.execute(MxRuntimeConnector.java:66) at com.mendix.util.classloading.Runner.doRunUsingClassLoaderOf(Runner.java:32) at com.mendix.externalinterface.connector.MxRuntimeConnector.processRequest(MxRuntimeConnector.java:71) at com.mendix.core.impl.MxRuntimeImpl.processRequest(MxRuntimeImpl.java:719) at com.mendix.m2ee.appcontainer.server.handler.RuntimeHandler.handle(RuntimeHandler.java:41) at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) at org.eclipse.jetty.server.Server.handle(Server.java:368) at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489) at org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53) at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:953) at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1014) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:861) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240) at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72) at org.eclipse.jetty.server.nio.BlockingChannelConnector$BlockingChannelEndPoint.run(BlockingChannelConnector.java:298) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) at java.lang.Thread.run(Unknown Source)
asked
2 answers
3

Hi Dirk, could you try to upload to S3 when running the Mendix project locally? In that way we can check if your issue arises only when running on Pivotal or that it's more generic.

To run a Mendix project locally with S3 enabled, you can add the following to the custom settings of your Mendix project: key: com.mendix.core.StorageService, value: com.mendix.storage.s3. After that also add your S3 settings to the custom settings. See the settings information here for which keys and values to use.

If you run the Mendix application with these custom settings configured it will use S3 instead of the local file system. Can you verify if you get the same error when uploading a file in this setup?

answered
1

Dirk, I found this thread on StackOverflow with a similar issue

I also found the source on GitHub which shows you where this particular error is generated.

It's not a complete answer but perhaps gets you moving in the right direction. Good luck! If you want to share your java code, I'm sure we could give it a closer look.

answered