If you are looking for open source alternative of Matlab then try gnu octave . according to octave official site :

GNU Octave is a high-level interpreted language, primarily intended for numerical computations. It provides capabilities for the numerical solution of linear and nonlinear problems, and for performing other numerical experiments. It also provides extensive graphics capabilities for data visualization and manipulation. Octave is normally used through its interactive command line interface, but it can also be used to write non-interactive programs. The Octave language is quite similar to Matlab so that most programs are easily portable.

Installing Octave on Mac

follow these simple steps to get and running octave on your mac osx ( i am using mavericks 10.9 in this case ) .
Lest first install some requirements .

Download Xcode 

and install xcode command line tools 

then open terminal and type

ruby -e "$(curl -fsSL https://raw.github.com/Homebrew/homebrew/go/install)"

now update and tap homebrew science

brew update 
brew tap homebrew/science

now install gcc and automake ( just make sure its compile )

brew tap homebrew/versions
brew install gcc49 
brew install gfortran 
brew install octave

if you having trouble in mavericks use that patch ( it worked for me )

brew install https://raw.github.com/Homebrew/homebrew-science/3c3fe3baaf926437f750f65456769c124d6be8e1/octave.rb --env=std