Convex Optimization

This review is written from the perspective of a mathematically capable outsider to the field of Operations Research and Optimization that does research in statistics & machine learning. I have found that it's very difficult to figure out how to self-study these topics in general, so I'm going to include context & other resources I have found useful to highlight the importance and quality of this book. In my opinion, the landscape of OR and Optimization have to be understood to really appreciate it.When I initially picked up Boyd & Vanderberghe, I got very frustrated with it. I would recommend going through introductory optimization material first (I went through "Managerial Decision Modeling with Spreadsheets" by Balakrishnan et al, which is excellent), and be comfortable with the basics of linear algebra and calculus, otherwise this book will seem more inaccessible than it actually is. I have also heard that Stephen Boyd's lectures on YouTube are an excellent supplement to the book and make the introduction far more gentle.This book is a field manual for learning to recognize convex problems "in the wild": meaning, given an optimization problem, how do we make it easier to solve by making one or more parts of the problem convex? This process is known as "convex relaxation" and is one of the fundamental skills required for research in topics like statistics & machine learning, and probably many other fields as well. Because of this focus, the book strikes a very important middle ground between the highly applied side of case studies and model building, and the highly theoretical side of algorithms and convex analysis. It provides just enough theory to be able to work with problems, and just enough intuition on algorithms to know what's going on behind the scenes.I don't know why, but most treatments I have seen of Optimization/Operations Research is presented with too much of a focus on either theory/algorithms, or case studies with barely any focus on the former. After reading this text and doing the exercises (for which the solutions are freely available online), the reader can expect to be able to formulate optimization problems, relax those problems to be more easily solvable, input the problems to an optimization solver like CVX (Matlab) or CVXPY (CVX, but in Python) and have an idea of what algorithms the solver will use to solve the problem.From what I have seen so far, I believe this text alone is sufficient for a good initial background to do statistics & machine learning research. However, I don't believe it's enough for people who want to focus on the difficulties of Operations Research model building, writing optimization algorithms, or on theoretical topics like existence of solutions, convergence proofs for algorithms, etc.For discussion on the challenges associated with building big optimization models, I recommend H. Paul Williams's "Model Building in Mathematical Programming" (2013)--multiple real-world problems, their model formulations, and solutions are presented, and it's just amazing practice for a budding analyst. I think the introductory chapters are definitely lacking though, and I recommend picking up "Managerial Decision Modeling with Spreadsheets", as I mentioned above.For learning how to write numerical optimization algorithms, it is hard to go wrong with Nocedal's and Wright's "Numerical Optimization" (2006). The presentation is, like this book, crystal clear, but again requires the reader to have a good grasp on the basics of optimization, linear algebra, and calculus. The entire focus of the book is on algorithms to solve various optimization problems (not just convex problems!), but because of its age, it will not contain material on some of the newer algorithms that have found use. I would recommend looking through (free) online courses such as Ryan Tibshirani's "Machine Learning 10-725", Stephen Boyd's EE364A/EE364B sequence, and Emmanuel Candes's Math 301.Unfortunately, I am not the right person to ask for book recommendations to learn the theory, but I have heard good things about books by Nesterov, Bertsekas, and Rockafellar. These are books that really demand the reader to have a background in mathematical analysis. From my limited searching, Bertsekas's books have solution manuals, have good geometric descriptions of the concepts, and are self-contained. The books "Convex Optimization Theory" (2009), "Convex Optimization Algorithms" (2015), and "Convex Analysis & Optimization" (2003) in particular might be a good place to start.

Quite simply, this is a wonderful text. Coupling this with Boyd's course at Stanford (the lecture videos, HWs, etc. are all available for free online), you're bound to learn quite a lot about optimization. But most importantly, you'll have an idea of when you can actually apply convex optimization to solve a problem that comes up in your particular field.My reasoning in giving it such praise is my preference for the rather unusual methodology it takes in introducing you to optimization. Most books I have seen on linear programming or non-linear programming tackle a few standard problems, introduce what is necessary in terms of definitions and proofs, and then focus on the algorithms that solve these standard problems (conjugate gradient et. al.), how they work, their pitfalls, etc. While this is undoubtedly useful material (which Boyd does cover for a good deal in the final chapters), the simple fact of the matter is these algorithms are available as standard methods in optimization packages (which are abstracted from the user), and unless you are actually going into developing, implementing and tweaking algorithms, this quite honestly is useless.What this book attempts to do, and does very well in my opinion, is to teach you to recognize convexity that's present in problems that are first glance appear to be so incredibly removed from optimization that you might never consider it. This book spends the first 100 pages or so just devoted to building a "calculus" of convexity, if you will, so that you know through what operations convexity is preserved, and you develop intuition as to the potential to use convex optimization in problems in your particular field or application. As such, the first part of the books is focused on building up the skill set, the second part to applications of convex programming, and only the third to the actual algorithms.A word of warning: some of the explanations (especially in Chapter 4 which focuses on types of convex programs and equivalence of programs) are very general, which won't be satisfying to certain readers who need solid examples to reinforce the concepts. Also, a lot of the material can be quite challenging, requiring a bit of mental gymnastics. However, if you are accompanying your study with the problems at the end of each chapter, you're certain to get practice and demystify the concepts.In sum, all things considered, a great text.

I just finished the reading of the entire book, and I feel that I am blessed that I have done this! The reading of this book is pure joy. I found all chapters easy to follow (whatever you needed was in the book, no need to look elsewhere) except possible Chapter 7 (statistical estimation). I found Chapter 10 and the 2nd half of chapter 11 the most difficult to read, but persistence will reward you. This book does not only teaches you Convex Optimization, but it also teaches you Matrix Analysis what-u-need-to-know, and after going through the examples, one gains enough knowledge about Matrix analysis to be able to apply it. There is also a free solution manual for the exercises of the book that someone should have handy because many exercises expand on the understanding of the concepts.

I've spoken with Dr. Stephen Boyd. Very nice man and very helpful instructor. I'm taking a free-online course to supplement this text and get a better understanding. There are other optimization books and techniques, and this one is a good as any. Also, there are some Matlab tools that can be downloaded for free to use with this book. Highly recommend.

If you are a researcher or engineer in the field of optimization, you must read this book and have it on your desk at all times. Very lucid, clear chapters. I read it cover to cover, and keeping returning to it for refreshers and new insights.

This material is somewhat tangential to my research, but I learned a ton by reading it. Very well organized.For example, here is a problem I was working on. For a given matrix A, find vectors a and b such that1. |A| <= ab^T, (outer product) and2. a^Tb (inner product) is a minimum. Convex Optimization showed me how to convert this into a CO problem