Python
The problem you're experiencing comes from the fact that array.append, as implemented in NumPy, actually creates a new copy of the array every time called, which must allocated in the memory completely new, especially if called often and on large arrays this should definitely be avoided. There are two solutions to solve this issue:
1. If you don't know how big the arrays are going to be, you can use python lists during the processing and then convert them to NumPy arrays afterwards. Python lists alter in memory and are therefore much more performant for using append. The downside of this is that you have larger memory usage, as you can't use NumPys more efficient data types.
2. If you know how big the arrays are going to be you can preallocate the whole array, by using the zeros method of NumPy and create sort of an empty array which you then add the values to as you progress. This may requires changing of your processing algorithm and therefore could be slower, but it can be way more memory efficient.
it seems that the reason why Docker uses much more memory is that it does not allow the operating system to use virtual memory (swap memory). What I didn't see in the original debugging, is that the python program consumes memory of several GB, but windows has automatically moved another several GB of memory to virtual memory to save RAM space. This is however not enabled in docker environment, so the only RAM available to the docker is the one that is specified in the set up phase.
Taggings:
The answer is: it depends.
It depends on the programming language you are using and partially on the database you are using.
Most of the databases are services running on a web service. We can communicate with them using a special protocol, while we know the address and the port of the server where it runs. There are also databases stored in files, where we specify the target file.
Most of the programming lanugages and frameworks provide a "Connector" for databases, which allows us to communicate with the database. Of course you need to check if your programming language has a driver for the database type you want to connect to. Some frameworks or programming languaes also offer a higher level control structures called object-relational mapping, where the user can easily read or write some objects defined in the project.
There are many different possibilities to answer this question. We will go top down beginning with the browser.
Most of the browsers support the html input tag of type "email" and most of them would automatically warn the user about a wrong format of the address. However there are many standards of email address pattern. Furthermore the user might disable the browser validation, so we should not rely on this.
The second station is the javascript of the browser. Probably the easiest way to validate the email address is to use a regular expression an match the input against this expression. However the user may disable the javascript or send a request in other way than browser. Therefore it's absolutely necessary to validate the email address on the server side. This might be done by a regular expression, but some more sophisticated systems would check the MX record of the domain given in the email address, to be sure it might be a real address of a real mail server.
Use automated UI Test, which is possible with the Selenium library and a browser driver. The browser driver allows us to run actions in a browser while writing code. The Selenium library is an interface to connect many browsers with the same code base. So now we are able to code a simple test, where a robot is automatically clicking through the website and trying out the login functionality. This technique may also apply to test other important and high priority UI features.
One solution is to use task queues. In Django it is easy to implement a task queue with a third-party library called "huey". It allows developers to specify when to execute a task using annotations:
@huey.periodic_task(crontab(minute='*'))
def print_time():
print(datetime.now())
This task would be executed every minute. Apart from scheduled tasks it also offers the option to execute tasks in the future.
Taggings:
