As an alternative to MATLAB/Octave, may I suggest using Python with Jupyter Notebooks?
For exploratory/analysis/prototyping work, I have not found a better platform and ecosystem. These notebooks are really powerful feature-wise, you can mix Markdown notes/documentation with code blocks with data visualization natively, which is a no brainer when doing your research work.
How does Jupyter compare with an environment like Spyder, which basically mimics the MATLAB approach? Can I inspect variables, run arbitrary commands against them in a REPL?
I don't think it's an issue of having some general language for doing math work. My understanding is that MATLAB specifically has a ton of built in functionality for filter design and coefficient calculation.
For exploratory/analysis/prototyping work, I have not found a better platform and ecosystem. These notebooks are really powerful feature-wise, you can mix Markdown notes/documentation with code blocks with data visualization natively, which is a no brainer when doing your research work.