![]() ![]() The EC2 instance just “steals” said IP on boot via an AWS CLI call in a bash script. Our customers require a static IP address, which we implemented via an ElasticIP. Thus, our solution was an ASG of size 1, which at least allows for an unhealthy instance to be replaced, and we selected a powerful enough instance to handle peak load. FTP is an incredibly hard to proxy protocol though. Ideally, we would like a dynamic Auto Scaling Group with a flexible amount of FTP server EC2s, load-balanced by an AWS ELB. The FTP server itself is a straightforward EC2 instance with a piece of software on it. Initial Implementation using Storage Gateway Architecture Overview This is kind of by design, as parts of the system are intended to be run on-premise, out of reach of AWS automation. ![]() It does not support CloudFormation or AWS CLI to create. The biggest downside of Storage Gateway is that it requires manual steps to set it up. This provided the resilient and AWS-native storage backend we were looking for.Įvent queues can easily be attached to the S3 bucket and any further systems can readily access the same S3 bucket simultaneously. As a result, Storage Gateway basically allowed us to mount an S3 bucket as backing storage into our FTP server EC2 instance. Whilst it is an appliance designed to run on on-premise hardware to make AWS storage available in the local network, one can run it on AWS too. Storage Gateway allows accessing cloud storage like S3 (and others) via NFS (or SMB and even iSCSI). Also it would not work readily with Lambda. It would also mean running the system that translates FTP uploads to our HTTP API format in an EC2 instance in order to get access to the fileshare. However, we would not get notified of incoming files – unless we would create an SQS queue that the FTP server puts the path of incoming files into. We could mount this into the EC2 instance running the FTP server and simultaneously any other EC2 instance to share data between them. One would need an additional piece of integration software on the same instance to get files out of the volume. Whilst this approach provides a data store that outlives a failing instance, it is not easily accessible by other AWS services outside the instance it runs on. EBS – Elastic Block StorageĪ classic approach for storage that persists beyond the lifecycle of a single EC2 instance is to attach an EBS volume. However, we now need to provide storage that is both resilient against instance failure and integrates nicely with other AWS services. The instance in turn is within an ASG to be resilient against instance failure. So, this approach would also have required further workarounds.Īs a result, we had to run our own FTP server in an EC2 instance. But even if that would have been an option at the time, the FTP version is only available inside a VPC (including e.g. Later on Amazon has added “Transfer for FTP”. And sadly, the interface we must provide has to have a non-S FTP option. At the time of evaluating the service though, “Transfer for SFTP” was the only available option. Not only does that sound too good to be true, it would also make for a rather short and anti-climactic article. Options Considered TransferĪ fully managed FTP server that is seamlessly backed by an S3 bucket. We share these key lessons on this article. Making it an “AWS-native” to integrate it with the new HTTP API using AWS infrastructure was not exactly straightforward and many lessons were learnt. The challenges in this setup lie not merely in running a resilient FTP server in AWS. So, whilst the files are rather small, usually less than 10 kB, each transfer causes a considerable amount of compute, from handling the FTP connection to notifying several involved subsystems. Under the hood said FTP server should trigger transformation of uploaded files to the format of the new API and then, either push them to the new API, or use a notification mechanism like SQS to mark them for pickup from a shared location. At the same time, the system has to hold historical data, which spans across millions of files. These are comprised of over 50,000 uploads, usually in the form of listings, and up to 50,000 downloads, usually in the form of customer-facing logs. Performance-wise, the system has to handle an average of 100,000 transfers per day. To get out of the data centre and avoid running two systems, we decided to operate an FTP server in AWS. The legacy FTP solution however was still running in our data centre, which was about to shutdown. At the same time, we need to support the legacy alternative in the form of an FTP server.įor quite a while REA has been building its systems in AWS. In the case at hand, as of beginning of 2020 REA is providing a new HTTP API for uploading listings. When re-designing customer-facing APIs, having a smooth migration path is rarely an avoidable topic.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |