We've been packaging pandas in a lambda which is used to perform some calculations, but being a 50 MB zip file makes cold starts of about 6-8 secs. We're lucky that the service has little use, thus our way to workaround it is by having a lambda warmer which is run every 5 minutes and invokes N pandas lambdas. I'd be very interested in knowing if Layers has some feature to avoid this kind of issue.
We had the same cold start problem and couldn’t find a way to reliably keep things warm. For instance, Lambda would often spawn a new instance rather than re-use the warm one.
In the end, we came to the conclusion that Amazon is smart and won’t let you hack together the equivalent of a cheaper EC2.
I don't think it's deliberately so, just that developing a solution requires scheduling and routing to cooperate. Normally they're considered by separate systems. As your execution pool expands, this problem becomes worse, not better.
On the other hand, their incentive to solve the problem is relatively weak vs an on-premise alternative.
If I were doing this today, I would prototype the problem in Python and after realising the startup penalty, would rewrite it in D's Mir [1] or Nim's ArrayMancer [2].
Life on a lambda is too short to pay 6-8 second startup penalty over and over millions of time.
Our problem is that we have a team of data scientists who are familiar with Python, plus a decent set of custom tools written in it, so changing languages isn't an option
that's often the current explanation for continued use of Pyhton and R.
Often it is a sign that the problem is not "big" enough (eg: not crunching truly large data sets) OR data science team gets disproportionate amount of goodwill (thus money) to spend on its foibles. :)
How did you get the zip down to 50MB. I was under the impression that pandas+numpy was closer to 300MB and bumped up against AWS size limits. I was considering building some hacked together thing with S3
I came to this thread specifically to find out about numpy and pandas on lambda.
We've been running a stripped down version of numpy + scipy + matplotlib in lambda. We'd build the environment in a docker container with Amazon linux, manually remove unneeded shared objects and then rezip the local environment before uploading to s3.
When I worked on this I used this article as a reference: https://serverless.com/blog/serverless-python-packaging/ and also ended up with a huge image. What that article didn't mention is that the virtual environment folder should be excluded in the serverless config file, as the runtime is provided by boto3. So adding:
package:
exclude:
- venv/
would reduce the size considerably (to 50 MB in my case)
It is in our case. This is a service which is very seasonal, so it may be used during a couple of days each month only. Having a bunch of instances mostly idle would definitely be more expensive
From one of the video comments: "They (Maria João, conductor, and orchestra) had recorded 6 months earlier three Mozart concerti. This lunch concert was the rehearsal for the evening performance. Since they had rehearsed to prepare the recording six months earlier, this lunch concert was the only rehearsal. She came prepared to play K. 467 and was caught off guard by K. 466 (although it has been in her repertory for years)."
I've found OneTab to be a better idea than Suspender personally, plus it lets me look back at all the stuff I did have open during a session for quick recall.
It isn't great, and they did have a leak, but it is the best option if you're working across platforms (including your phone). And peering options are at least there. I've considered switching out, but have been relatively happy.
Website is a complete mess and the page you refer to was probably about the previous model which was withdrawn. Anyway, after some search I managed to find it here in Spain and even order one (they have a nice discount these days due to Black Friday). This one should make it for you: http://www.dell.com/de/unternehmen/p/xps-13-9360-laptop/pd?o...
I'm in a similar situation to yours and been reading some positive reviews about this https://slimbook.es/
They seem to be very responsive in terms of doubts, questions, etc. but as most small companies their main disadvantage is brand awareness when compared to Dell or Apple. But of course this is greatly compensated by a cheaper price.
